[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:02 dist-test-slave-l766 krb5kdc[4416](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:02 dist-test-slave-l766 krb5kdc[4416](info): set up 2 sockets
Sep 24 15:23:02 dist-test-slave-l766 krb5kdc[4416](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 15:23:03.412539 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:03 dist-test-slave-l766 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:03.532056 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39327
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:03.613090 4429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:03.613358 4429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:03.613394 4429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:03.614599 4429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:03.614672 4429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:03.614696 4429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:03.614719 4429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:03.614748 4429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:03.616240 4429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:03.616524 4429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:03.616750 4429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:03.621997 4434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:03.622012 4429 server_base.cc:1034] running on GCE node
W20260924 15:23:03.621969 4437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:03.621969 4435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:03.622511 4429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:03.622836 4429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:03.623980 4429 hybrid_clock.cc:648] HybridClock initialized: now 1790263383623966 us; error 27 us; skew 500 ppm
Sep 24 15:23:03 dist-test-slave-l766 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263383, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:03.629802 4429 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:03.630292 4429 webserver.cc:460] Webserver started at http://127.4.78.126:42583/ using document root <none> and password file <none>
I20260924 15:23:03.630527 4429 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:03.630573 4429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:03.630743 4429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:03.631804 4429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "216ce6fa9c344b40b091594c3a861cdd"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "1449f744caadf98734de1e07ffe3f183"
server_key_iv: "af6e0637b7af376462df7c3970397725"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.632174 4429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "216ce6fa9c344b40b091594c3a861cdd"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "1449f744caadf98734de1e07ffe3f183"
server_key_iv: "af6e0637b7af376462df7c3970397725"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.633468 4429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:03.634279 4444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:03.634460 4429 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:03.634567 4429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "216ce6fa9c344b40b091594c3a861cdd"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "1449f744caadf98734de1e07ffe3f183"
server_key_iv: "af6e0637b7af376462df7c3970397725"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.634651 4429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:03.657250 4429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:03.659363 4429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:03.659533 4429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:03.663604 4429 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:39327
I20260924 15:23:03.663657 4497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:39327 every 8 connection(s)
I20260924 15:23:03.664036 4429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:03.664923 4498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:03.666684 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 4429
I20260924 15:23:03.666767 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:03.667407 4409 external_mini_cluster.cc:1428] Setting key 3e63dd6ee087d3ad1ef4342dd5c9dba9
I20260924 15:23:03.667430 4498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd: Bootstrap starting.
I20260924 15:23:03.668105 4498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:03.668463 4498 log.cc:824] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:03.669203 4498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd: No bootstrap required, opened a new log
I20260924 15:23:03.670902 4498 raft_consensus.cc:348] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216ce6fa9c344b40b091594c3a861cdd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39327 } }
I20260924 15:23:03.671062 4498 raft_consensus.cc:374] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:03.671109 4498 raft_consensus.cc:729] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216ce6fa9c344b40b091594c3a861cdd, State: Initialized, Role: FOLLOWER
I20260924 15:23:03.671243 4498 consensus_queue.cc:260] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216ce6fa9c344b40b091594c3a861cdd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39327 } }
I20260924 15:23:03.671336 4498 raft_consensus.cc:388] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:03.671373 4498 raft_consensus.cc:482] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:03.671425 4498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:03.672231 4498 raft_consensus.cc:504] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216ce6fa9c344b40b091594c3a861cdd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39327 } }
I20260924 15:23:03.672360 4498 leader_election.cc:302] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 216ce6fa9c344b40b091594c3a861cdd; no voters:
I20260924 15:23:03.672556 4498 leader_election.cc:288] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:03.672581 4503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:03.672686 4503 raft_consensus.cc:686] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [term 1 LEADER]: Becoming Leader. State: Replica: 216ce6fa9c344b40b091594c3a861cdd, State: Running, Role: LEADER
I20260924 15:23:03.672819 4498 sys_catalog.cc:564] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:03.672786 4503 consensus_queue.cc:237] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216ce6fa9c344b40b091594c3a861cdd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39327 } }
I20260924 15:23:03.673321 4504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 216ce6fa9c344b40b091594c3a861cdd. Latest consensus state: current_term: 1 leader_uuid: "216ce6fa9c344b40b091594c3a861cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216ce6fa9c344b40b091594c3a861cdd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39327 } } }
I20260924 15:23:03.673417 4504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:03.673321 4505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "216ce6fa9c344b40b091594c3a861cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "216ce6fa9c344b40b091594c3a861cdd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39327 } } }
I20260924 15:23:03.673599 4505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:03.674382 4507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:03.674863 4507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:03.676258 4507 catalog_manager.cc:1347] Generated new cluster ID: 52d0a56a08c7495a9b9d644ee64efc9f
I20260924 15:23:03.676316 4507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:23:03 dist-test-slave-l766 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:03.685703 4507 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:03.686328 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:03.668316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51734 (local address 127.4.78.126:39327)
0924 15:23:03.668495 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:03.668506 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:03.668783 (+ 277us) server_negotiation.cc:407] Connection header received
0924 15:23:03.670889 (+ 2106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:03.670895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:03.670939 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:03.671077 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:03.672768 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:03.674380 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:03.676310 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:03.676926 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:03.677045 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:03.681509 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:03.681513 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:03.681514 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:03.681525 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:03.684178 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:03.684675 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:03.684678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:03.684678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:03.684800 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:03.685114 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:03.685115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:03.685115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:03.685359 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:03.685532 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:03.685957 (+ 425us) server_negotiation.cc:299] Negotiation successful
0924 15:23:03.686094 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260924 15:23:03.686424 4507 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:03.691634 4507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 216ce6fa9c344b40b091594c3a861cdd: Generated new TSK 0
I20260924 15:23:03.691813 4507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:03.718290 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39327
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:03.801928 4524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:03.802165 4524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:03.802207 4524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:03.803342 4524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:03.803406 4524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:03.803452 4524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:03.804898 4524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4524
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:03.805176 4524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:03.805414 4524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:03.811527 4530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:03.811717 4524 server_base.cc:1034] running on GCE node
W20260924 15:23:03.811784 4532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:03.811553 4529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:03.812021 4524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:03.812278 4524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:03.813432 4524 hybrid_clock.cc:648] HybridClock initialized: now 1790263383813407 us; error 37 us; skew 500 ppm
Sep 24 15:23:03 dist-test-slave-l766 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263383, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:03.816262 4524 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:03.816749 4524 webserver.cc:460] Webserver started at http://127.4.78.65:40885/ using document root <none> and password file <none>
I20260924 15:23:03.816990 4524 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:03.817054 4524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:03.817207 4524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:03.818310 4524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "9b191f26396043e686a8e13e0aaa522c"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "7b565c0f314047c5097007c9af7d735e"
server_key_iv: "77c5f994c67338a4f5e57b4e63f982ed"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.818670 4524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "9b191f26396043e686a8e13e0aaa522c"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "7b565c0f314047c5097007c9af7d735e"
server_key_iv: "77c5f994c67338a4f5e57b4e63f982ed"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.819968 4524 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:03.820667 4539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:03.820858 4524 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:03.820955 4524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "9b191f26396043e686a8e13e0aaa522c"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "7b565c0f314047c5097007c9af7d735e"
server_key_iv: "77c5f994c67338a4f5e57b4e63f982ed"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.821031 4524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:03.840183 4524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:03.842120 4524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:03.842291 4524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:03.842538 4524 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:03.842872 4524 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:03.842926 4524 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:03.842965 4524 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:03.842988 4524 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:03.848820 4524 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:42139
I20260924 15:23:03.848924 4652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:42139 every 8 connection(s)
I20260924 15:23:03.849233 4524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:03.853495 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 4524
I20260924 15:23:03.853600 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:03.853770 4409 external_mini_cluster.cc:1428] Setting key 517c76251b6a6def235a2de385575974
Sep 24 15:23:03 dist-test-slave-l766 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263383, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:03.859423 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:03.850019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:34327 (local address 127.4.78.126:39327)
0924 15:23:03.850085 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:03.850087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:03.850304 (+ 217us) server_negotiation.cc:407] Connection header received
0924 15:23:03.850849 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:03.850851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:03.850868 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:03.850919 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:03.851889 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:03.852451 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:03.853495 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:03.853782 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:03.853911 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:03.856426 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:03.856428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:03.856428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:03.856437 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:03.857974 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:03.858389 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:03.858391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:03.858391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:03.858449 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:03.858727 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:03.858728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:03.858729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:03.858852 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:03.858930 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:03.859240 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 15:23:03.859331 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:23:03.859637 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:03.850028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39327 (local address 127.4.78.65:34327)
0924 15:23:03.850188 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:03.850224 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:03.850735 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:03.850993 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:03.850997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:03.851572 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:03.851785 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:03.851790 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:03.852561 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:03.852562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:03.853389 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:03.853391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:03.853929 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:03.853931 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:03.854067 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:03.854686 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:03.854695 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:03.856337 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:03.858070 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:03.858077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:03.858080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:03.858312 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:03.858540 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:03.858541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:03.858541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:03.858654 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:03.858920 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:03.858927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:03.859100 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 15:23:03.859217 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 15:23:03.859380 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260924 15:23:03.860148 4653 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39327
I20260924 15:23:03.860271 4653 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:03.860476 4653 heartbeater.cc:511] Master 127.4.78.126:39327 requested a full tablet report, sending...
I20260924 15:23:03.861092 4461 ts_manager.cc:194] Registered new tserver with Master: 9b191f26396043e686a8e13e0aaa522c (127.4.78.65:42139)
I20260924 15:23:03.862123 4461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:34327
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:03.881084 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39327
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:03.964222 4658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:03.964417 4658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:03.964440 4658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:03.965588 4658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:03.965628 4658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:03.965654 4658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:03.967121 4658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:03.967308 4658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:03.967480 4658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:03.972461 4666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:03.972595 4663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:03.972461 4664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:03.973428 4658 server_base.cc:1034] running on GCE node
I20260924 15:23:03.973630 4658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:03.973910 4658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:03.975073 4658 hybrid_clock.cc:648] HybridClock initialized: now 1790263383975055 us; error 33 us; skew 500 ppm
Sep 24 15:23:03 dist-test-slave-l766 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263383, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:03.977530 4658 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:03.978119 4658 webserver.cc:460] Webserver started at http://127.4.78.66:33147/ using document root <none> and password file <none>
I20260924 15:23:03.978374 4658 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:03.978417 4658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:03.978562 4658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:03.979732 4658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "da6e30afd5274e1f954411ab78c98f4f"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "a8ede1b3099303bf73fba146d33618bc"
server_key_iv: "27051cf81a13962c117367165b9a59c2"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.980087 4658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "da6e30afd5274e1f954411ab78c98f4f"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "a8ede1b3099303bf73fba146d33618bc"
server_key_iv: "27051cf81a13962c117367165b9a59c2"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.981544 4658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:03.982347 4674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:03.982563 4658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:03.982681 4658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "da6e30afd5274e1f954411ab78c98f4f"
format_stamp: "Formatted at 2026-09-24 15:23:03 on dist-test-slave-l766"
server_key: "a8ede1b3099303bf73fba146d33618bc"
server_key_iv: "27051cf81a13962c117367165b9a59c2"
server_key_version: "encryptionkey@0"
I20260924 15:23:03.982767 4658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:03.991823 4658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:03.993613 4658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:03.993789 4658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:03.994112 4658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:03.994431 4658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:03.994485 4658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:03.994520 4658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:03.994547 4658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:04.000854 4658 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:42619
I20260924 15:23:04.000909 4789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:42619 every 8 connection(s)
I20260924 15:23:04.001243 4658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:04.006161 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 4658
I20260924 15:23:04.006314 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:04.006507 4409 external_mini_cluster.cc:1428] Setting key 82c7cb9923b9299559d18b6cf91c3296
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263383, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:04.011296 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:04.002045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:52669 (local address 127.4.78.126:39327)
0924 15:23:04.002113 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:04.002115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:04.002340 (+ 225us) server_negotiation.cc:407] Connection header received
0924 15:23:04.002887 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:04.002888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:04.002901 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:04.002956 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:04.003945 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.004470 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.005502 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.005725 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.005805 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:04.008305 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:04.008307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:04.008307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:04.008316 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:04.009867 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:04.010301 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:04.010303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:04.010303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:04.010367 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:04.010624 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:04.010625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:04.010626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:04.010747 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:04.010825 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:04.011098 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 15:23:04.011206 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:23:04.011473 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:04.002030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39327 (local address 127.4.78.66:52669)
0924 15:23:04.002216 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:04.002256 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:04.002781 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:04.003032 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:04.003036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:04.003626 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:04.003841 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:04.003846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.004572 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.004574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:04.005407 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:04.005410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.005827 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.005829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:04.005983 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:04.006606 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:04.006618 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:04.008203 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:04.009968 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:04.009975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:04.009978 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:04.010209 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:04.010444 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:04.010444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:04.010445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:04.010557 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:04.010816 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:04.010822 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:04.010967 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 15:23:04.011077 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:23:04.011234 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260924 15:23:04.012037 4790 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39327
I20260924 15:23:04.012162 4790 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:04.012377 4790 heartbeater.cc:511] Master 127.4.78.126:39327 requested a full tablet report, sending...
I20260924 15:23:04.012816 4461 ts_manager.cc:194] Registered new tserver with Master: da6e30afd5274e1f954411ab78c98f4f (127.4.78.66:42619)
I20260924 15:23:04.013259 4461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:52669
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:04.032925 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39327
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:04.115777 4795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:04.115962 4795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:04.115988 4795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:04.117118 4795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:04.117162 4795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:04.117189 4795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:04.118563 4795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39327
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:04.118737 4795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:04.118903 4795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:04.123670 4803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:04.123782 4795 server_base.cc:1034] running on GCE node
W20260924 15:23:04.123732 4801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:04.123685 4800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:04.124083 4795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:04.124389 4795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:04.125532 4795 hybrid_clock.cc:648] HybridClock initialized: now 1790263384125516 us; error 30 us; skew 500 ppm
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263384, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:04.128060 4795 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:04.128474 4795 webserver.cc:460] Webserver started at http://127.4.78.67:42135/ using document root <none> and password file <none>
I20260924 15:23:04.128712 4795 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:04.128757 4795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:04.128898 4795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:04.129972 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "e87775bf741d4bd9925d4a20f1cb46f6"
format_stamp: "Formatted at 2026-09-24 15:23:04 on dist-test-slave-l766"
server_key: "7dcfe2d3356b287a309f36033dae5a83"
server_key_iv: "d4c973e07c929a0a750e7953f072670c"
server_key_version: "encryptionkey@0"
I20260924 15:23:04.130317 4795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "e87775bf741d4bd9925d4a20f1cb46f6"
format_stamp: "Formatted at 2026-09-24 15:23:04 on dist-test-slave-l766"
server_key: "7dcfe2d3356b287a309f36033dae5a83"
server_key_iv: "d4c973e07c929a0a750e7953f072670c"
server_key_version: "encryptionkey@0"
I20260924 15:23:04.131598 4795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:23:04.132299 4810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:04.132465 4795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:04.132567 4795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "e87775bf741d4bd9925d4a20f1cb46f6"
format_stamp: "Formatted at 2026-09-24 15:23:04 on dist-test-slave-l766"
server_key: "7dcfe2d3356b287a309f36033dae5a83"
server_key_iv: "d4c973e07c929a0a750e7953f072670c"
server_key_version: "encryptionkey@0"
I20260924 15:23:04.132638 4795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:04.151662 4795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:04.153939 4795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:04.154124 4795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:04.154352 4795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:04.154649 4795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:04.154708 4795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:04.154747 4795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:04.154775 4795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:04.160792 4795 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:34919
I20260924 15:23:04.160862 4924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:34919 every 8 connection(s)
I20260924 15:23:04.161190 4795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263384, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:04.167786 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 4795
I20260924 15:23:04.167904 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:04.168088 4409 external_mini_cluster.cc:1428] Setting key 57e5c8f91f4102501ab51c29178470a9
I20260924 15:23:04.170295 4500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:04.161956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:58613 (local address 127.4.78.126:39327)
0924 15:23:04.162016 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:04.162018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:04.162189 (+ 171us) server_negotiation.cc:407] Connection header received
0924 15:23:04.162668 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:04.162669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:04.162678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:04.162729 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:04.163600 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.164076 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.165046 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.165245 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.165295 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:04.167270 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:04.167271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:04.167272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:04.167278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:04.168962 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:04.169454 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:04.169455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:04.169455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:04.169519 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:04.169736 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:04.169737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:04.169737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:04.169898 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:04.169956 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:04.170170 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 15:23:04.170231 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:23:04.170447 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:04.161914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39327 (local address 127.4.78.67:58613)
0924 15:23:04.162073 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:04.162112 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:04.162586 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:04.162789 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:04.162792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:04.163345 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:04.163526 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:04.163531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.164163 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.164164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:04.164973 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:04.164975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.165316 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.165317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:04.165419 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:04.165876 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:04.165891 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:04.167184 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:04.169070 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:04.169077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:04.169079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:04.169375 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:04.169581 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:04.169581 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:04.169581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:04.169683 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:04.169950 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:04.169955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:04.170061 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 15:23:04.170142 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:23:04.170258 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260924 15:23:04.170912 4925 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39327
I20260924 15:23:04.171000 4925 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:04.171182 4925 heartbeater.cc:511] Master 127.4.78.126:39327 requested a full tablet report, sending...
I20260924 15:23:04.171556 4461 ts_manager.cc:194] Registered new tserver with Master: e87775bf741d4bd9925d4a20f1cb46f6 (127.4.78.67:34919)
I20260924 15:23:04.171871 4461 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:58613
I20260924 15:23:04.180284 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263384, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263384, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:04.202534 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:04.195208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.4.78.65:42139)
0924 15:23:04.195378 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:04.195384 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:04.195394 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:04.195429 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:04.195431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:04.195445 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:04.195529 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:04.196050 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.196816 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.197559 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.197953 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.198033 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:04.200199 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:04.200206 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:04.200206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:04.200213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:04.201573 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:04.201912 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:04.201915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:04.201915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:04.201970 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:04.202121 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:04.202122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:04.202122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:04.202242 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:04.202362 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:04.202398 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 15:23:04.202461 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
W20260924 15:23:04.202913 4567 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52276
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4416](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4416](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:04.224026 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:04.217619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.4.78.65:42139)
0924 15:23:04.217683 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:04.217685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:04.217722 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:23:04.217764 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:04.217765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:04.217775 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:04.217890 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:04.218422 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.218886 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.219572 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:04.219760 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:04.219810 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:04.221944 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:04.221946 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:04.221946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:04.221952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:04.223030 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:04.223383 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:04.223384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:04.223384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:04.223432 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:04.223593 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:04.223593 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:04.223594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:04.223680 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:04.223804 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:04.223904 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:23:04.223966 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:23:04.224572 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 4524
I20260924 15:23:04.229579 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 4658
I20260924 15:23:04.234045 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 4795
I20260924 15:23:04.238236 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 4429
2026-09-24T15:23:04Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1884 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4939](info): set up 2 sockets
Sep 24 15:23:04 dist-test-slave-l766 krb5kdc[4939](info): commencing operation
krb5kdc: starting...
W20260924 15:23:05.276360 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:05 dist-test-slave-l766 krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:05.390447 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:45463
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:45463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:05.469482 4952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:05.469676 4952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:05.469702 4952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:05.470830 4952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:05.470868 4952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:05.470882 4952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:05.470894 4952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:05.470906 4952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:05.472293 4952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:45463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:45463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:05.472476 4952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:05.472630 4952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:05.477283 4957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:05.477341 4960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:05.477468 4952 server_base.cc:1034] running on GCE node
W20260924 15:23:05.477643 4958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:05.477936 4952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:05.478171 4952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:05.479365 4952 hybrid_clock.cc:648] HybridClock initialized: now 1790263385479354 us; error 39 us; skew 500 ppm
Sep 24 15:23:05 dist-test-slave-l766 krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263385, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:05.481775 4952 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:05.482214 4952 webserver.cc:460] Webserver started at http://127.4.78.126:39731/ using document root <none> and password file <none>
I20260924 15:23:05.482440 4952 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:05.482506 4952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:05.482652 4952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:05.483723 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "0f98095f8b774e41965e45c3ccb5a66e"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "ecfd4c23560d6a754eccfd42375e11a3"
server_key_iv: "0e317e3674787675ef40fdf6ab446653"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.484092 4952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "0f98095f8b774e41965e45c3ccb5a66e"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "ecfd4c23560d6a754eccfd42375e11a3"
server_key_iv: "0e317e3674787675ef40fdf6ab446653"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.485488 4952 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:05.486254 4967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:05.486418 4952 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:05.486522 4952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "0f98095f8b774e41965e45c3ccb5a66e"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "ecfd4c23560d6a754eccfd42375e11a3"
server_key_iv: "0e317e3674787675ef40fdf6ab446653"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.486593 4952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:05.516741 4952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:05.519310 4952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:05.519483 4952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:05.523494 4952 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:45463
I20260924 15:23:05.523536 5020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:45463 every 8 connection(s)
I20260924 15:23:05.523939 4952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:05.524859 5021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:05.525159 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 4952
I20260924 15:23:05.525254 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:05.525417 4409 external_mini_cluster.cc:1428] Setting key c6d766097c27405f64e6d7681d743b89
I20260924 15:23:05.527906 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e: Bootstrap starting.
I20260924 15:23:05.528497 5021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:05.528789 5021 log.cc:824] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:05.529487 5021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e: No bootstrap required, opened a new log
I20260924 15:23:05.531399 5021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f98095f8b774e41965e45c3ccb5a66e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45463 } }
I20260924 15:23:05.531575 5021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:05.531610 5021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f98095f8b774e41965e45c3ccb5a66e, State: Initialized, Role: FOLLOWER
I20260924 15:23:05.531736 5021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f98095f8b774e41965e45c3ccb5a66e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45463 } }
I20260924 15:23:05.531823 5021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:05.531854 5021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:05.531900 5021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 0 FOLLOWER]: Advancing to term 1
Sep 24 15:23:05 dist-test-slave-l766 krb5kdc[4939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:05.532434 5021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f98095f8b774e41965e45c3ccb5a66e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45463 } }
I20260924 15:23:05.532559 5021 leader_election.cc:302] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f98095f8b774e41965e45c3ccb5a66e; no voters:
I20260924 15:23:05.532753 5021 leader_election.cc:288] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:05.532785 5026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:05.532893 5026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [term 1 LEADER]: Becoming Leader. State: Replica: 0f98095f8b774e41965e45c3ccb5a66e, State: Running, Role: LEADER
I20260924 15:23:05.533028 5021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:05.533004 5026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f98095f8b774e41965e45c3ccb5a66e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45463 } }
I20260924 15:23:05.533350 5026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f98095f8b774e41965e45c3ccb5a66e. Latest consensus state: current_term: 1 leader_uuid: "0f98095f8b774e41965e45c3ccb5a66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f98095f8b774e41965e45c3ccb5a66e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45463 } } }
I20260924 15:23:05.533424 5026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:05.533363 5027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f98095f8b774e41965e45c3ccb5a66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f98095f8b774e41965e45c3ccb5a66e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45463 } } }
I20260924 15:23:05.533790 5027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:05.533975 5030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:05.534461 5030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:05.535995 5030 catalog_manager.cc:1347] Generated new cluster ID: e2700fe760134fe0a7c9ab65e9a21187
I20260924 15:23:05.536051 5030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:05.536855 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:05.525982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.4.78.126:45463)
0924 15:23:05.526155 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:05.526163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:05.526188 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:23:05.526616 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:05.526623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:05.526671 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:05.526791 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:05.527573 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.528501 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.529343 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.529854 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.529987 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:05.532445 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:05.532449 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:05.532450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:05.532462 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:05.534967 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:05.535529 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:05.535532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:05.535532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:05.535648 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:05.535892 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:05.535893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:05.535894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:05.536228 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:05.536380 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:05.536508 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 15:23:05.536652 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260924 15:23:05.548918 5030 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:05.549448 5030 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:05.559072 5030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f98095f8b774e41965e45c3ccb5a66e: Generated new TSK 0
I20260924 15:23:05.559283 5030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:05.586824 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45463
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:05.686450 5047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:05.686655 5047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:05.686681 5047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:05.687747 5047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:05.687777 5047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:05.687803 5047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:05.689275 5047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:05.689460 5047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:05.689642 5047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:05.694442 5052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:05.694468 5055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:05.694442 5053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:05.694679 5047 server_base.cc:1034] running on GCE node
I20260924 15:23:05.694893 5047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:05.695133 5047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:05.696301 5047 hybrid_clock.cc:648] HybridClock initialized: now 1790263385696290 us; error 43 us; skew 500 ppm
Sep 24 15:23:05 dist-test-slave-l766 krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263385, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:05.698758 5047 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:05.699126 5047 webserver.cc:460] Webserver started at http://127.4.78.65:33469/ using document root <none> and password file <none>
I20260924 15:23:05.699329 5047 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:05.699390 5047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:05.699503 5047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:05.700479 5047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "6577de42ffe84a15a710976a9fd9aca3"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "c56477f00f0dd25850301cb13a20da0b"
server_key_iv: "381dee09fceb8c26ab0ad75c9026cdd7"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.700778 5047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "6577de42ffe84a15a710976a9fd9aca3"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "c56477f00f0dd25850301cb13a20da0b"
server_key_iv: "381dee09fceb8c26ab0ad75c9026cdd7"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.702054 5047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:05.702723 5062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:05.702913 5047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:05.703001 5047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "6577de42ffe84a15a710976a9fd9aca3"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "c56477f00f0dd25850301cb13a20da0b"
server_key_iv: "381dee09fceb8c26ab0ad75c9026cdd7"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.703058 5047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:05.737106 5047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:05.739002 5047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:05.739117 5047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:05.739322 5047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:05.739660 5047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:05.739701 5047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:05.739724 5047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:05.739773 5047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:05.745622 5047 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:37125
I20260924 15:23:05.745679 5176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:37125 every 8 connection(s)
I20260924 15:23:05.746001 5047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:23:05 dist-test-slave-l766 krb5kdc[4939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263385, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:05.752586 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 5047
I20260924 15:23:05.752694 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:05.752910 4409 external_mini_cluster.cc:1428] Setting key ef4e5dda2527f8727a1a369b100af021
I20260924 15:23:05.755775 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:05.746720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:55313 (local address 127.4.78.126:45463)
0924 15:23:05.746783 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:05.746785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:05.746974 (+ 189us) server_negotiation.cc:407] Connection header received
0924 15:23:05.747913 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:05.747914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:05.747925 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:05.747984 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:05.748601 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.749104 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.750160 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.750357 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.750412 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:05.752568 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:05.752569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:05.752570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:05.752577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:05.754389 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:05.754792 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:05.754793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:05.754794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:05.754860 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:05.755163 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:05.755164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:05.755164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:05.755267 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:05.755321 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:05.755642 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 15:23:05.755713 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:23:05.756059 5179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:05.746665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45463 (local address 127.4.78.65:55313)
0924 15:23:05.746835 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:05.746879 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:05.747810 (+ 931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:05.748056 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:05.748059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:05.748333 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:05.748526 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:05.748530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.749194 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.749195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:05.750082 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:05.750084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.750429 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.750429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:05.750523 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:05.751054 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:05.751062 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:05.752471 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:05.754472 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:05.754479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:05.754481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:05.754715 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:05.754930 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:05.754931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:05.754932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:05.755084 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:05.755333 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:05.755338 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:05.755504 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 15:23:05.755625 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 15:23:05.755804 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260924 15:23:05.756608 5177 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45463
I20260924 15:23:05.756712 5177 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:05.756888 5177 heartbeater.cc:511] Master 127.4.78.126:45463 requested a full tablet report, sending...
I20260924 15:23:05.757476 4985 ts_manager.cc:194] Registered new tserver with Master: 6577de42ffe84a15a710976a9fd9aca3 (127.4.78.65:37125)
I20260924 15:23:05.758482 4985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:55313
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:05.776746 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45463
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:05.855746 5182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:05.855929 5182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:05.855952 5182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:05.857036 5182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:05.857069 5182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:05.857095 5182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:05.858484 5182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:05.858660 5182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:05.858815 5182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:05.863483 5188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:05.863493 5187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:05.863649 5182 server_base.cc:1034] running on GCE node
W20260924 15:23:05.863503 5190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:05.863912 5182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:05.864176 5182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:05.865329 5182 hybrid_clock.cc:648] HybridClock initialized: now 1790263385865317 us; error 22 us; skew 500 ppm
Sep 24 15:23:05 dist-test-slave-l766 krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263385, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:05.867792 5182 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:05.868196 5182 webserver.cc:460] Webserver started at http://127.4.78.66:43227/ using document root <none> and password file <none>
I20260924 15:23:05.868415 5182 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:05.868485 5182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:05.868636 5182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:05.869634 5182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "c8048a0d417a47bf92bd7ef451d73f99"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "8cfb3ec1edaa06cd59180605a92bc4ca"
server_key_iv: "c817adc09d4d152f0eeac732be968300"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.870013 5182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "c8048a0d417a47bf92bd7ef451d73f99"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "8cfb3ec1edaa06cd59180605a92bc4ca"
server_key_iv: "c817adc09d4d152f0eeac732be968300"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.871335 5182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:05.872023 5197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:05.872186 5182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:05.872289 5182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "c8048a0d417a47bf92bd7ef451d73f99"
format_stamp: "Formatted at 2026-09-24 15:23:05 on dist-test-slave-l766"
server_key: "8cfb3ec1edaa06cd59180605a92bc4ca"
server_key_iv: "c817adc09d4d152f0eeac732be968300"
server_key_version: "encryptionkey@0"
I20260924 15:23:05.872370 5182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:05.903676 5182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:05.906162 5182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:05.906340 5182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:05.906587 5182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:05.906955 5182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:05.907011 5182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:05.907056 5182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:05.907080 5182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:05.913033 5182 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:43405
I20260924 15:23:05.913076 5311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:43405 every 8 connection(s)
I20260924 15:23:05.913380 5182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:23:05 dist-test-slave-l766 krb5kdc[4939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263385, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:05.921600 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 5182
I20260924 15:23:05.921674 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:05.921888 4409 external_mini_cluster.cc:1428] Setting key a6d114ebc7802ce773322c2f8301eee0
I20260924 15:23:05.922971 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:05.914067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:44195 (local address 127.4.78.126:45463)
0924 15:23:05.914134 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:05.914171 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:05.914381 (+ 210us) server_negotiation.cc:407] Connection header received
0924 15:23:05.915347 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:05.915349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:05.915362 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:05.915411 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:05.916042 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.916550 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.917584 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.917791 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.917916 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:05.919903 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:05.919904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:05.919905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:05.919912 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:05.921350 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:05.921892 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:05.921894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:05.921894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:05.921955 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:05.922283 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:05.922284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:05.922284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:05.922414 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:05.922483 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:05.922831 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 15:23:05.922908 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:23:05.923247 5314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:05.914036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45463 (local address 127.4.78.66:44195)
0924 15:23:05.914239 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:05.914286 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:05.915229 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:05.915478 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:05.915482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:05.915730 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:05.915942 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:05.915947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.916647 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.916648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:05.917472 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:05.917475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:05.917933 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:05.917934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:05.918047 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:05.918460 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:05.918468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:05.919801 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:05.921469 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:05.921475 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:05.921478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:05.921770 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:05.922049 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:05.922050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:05.922051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:05.922201 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:05.922498 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:05.922503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:05.922684 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 15:23:05.922803 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 15:23:05.922976 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260924 15:23:05.923676 5312 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45463
I20260924 15:23:05.923781 5312 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:05.923991 5312 heartbeater.cc:511] Master 127.4.78.126:45463 requested a full tablet report, sending...
I20260924 15:23:05.924417 4985 ts_manager.cc:194] Registered new tserver with Master: c8048a0d417a47bf92bd7ef451d73f99 (127.4.78.66:43405)
I20260924 15:23:05.924784 4985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:44195
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:05.948364 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45463
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:06.032665 5317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:06.032893 5317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:06.032933 5317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:06.034494 5317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:06.034554 5317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:06.034598 5317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:06.036068 5317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:06.036346 5317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:06.036576 5317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:06.041347 5323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:06.041353 5325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:06.041345 5322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:06.041644 5317 server_base.cc:1034] running on GCE node
I20260924 15:23:06.041915 5317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:06.042151 5317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:06.043332 5317 hybrid_clock.cc:648] HybridClock initialized: now 1790263386043316 us; error 29 us; skew 500 ppm
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:06.045785 5317 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:06.046231 5317 webserver.cc:460] Webserver started at http://127.4.78.67:38241/ using document root <none> and password file <none>
I20260924 15:23:06.046478 5317 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:06.046546 5317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:06.046681 5317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:06.047698 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "6821e8fc96d94111838ac7a0b0f4136e"
format_stamp: "Formatted at 2026-09-24 15:23:06 on dist-test-slave-l766"
server_key: "e6dfb8a2545e3db8a8625d7cf13e2b69"
server_key_iv: "d428bc5ad5133ea5ee24afcdccf89095"
server_key_version: "encryptionkey@0"
I20260924 15:23:06.048061 5317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "6821e8fc96d94111838ac7a0b0f4136e"
format_stamp: "Formatted at 2026-09-24 15:23:06 on dist-test-slave-l766"
server_key: "e6dfb8a2545e3db8a8625d7cf13e2b69"
server_key_iv: "d428bc5ad5133ea5ee24afcdccf89095"
server_key_version: "encryptionkey@0"
I20260924 15:23:06.049388 5317 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:06.050177 5332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:06.050348 5317 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:06.050447 5317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "6821e8fc96d94111838ac7a0b0f4136e"
format_stamp: "Formatted at 2026-09-24 15:23:06 on dist-test-slave-l766"
server_key: "e6dfb8a2545e3db8a8625d7cf13e2b69"
server_key_iv: "d428bc5ad5133ea5ee24afcdccf89095"
server_key_version: "encryptionkey@0"
I20260924 15:23:06.050570 5317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:06.065335 5317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:06.067751 5317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:06.067924 5317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:06.068166 5317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:06.068542 5317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:06.068602 5317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:06.068645 5317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:06.068671 5317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:06.074563 5317 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:34103
I20260924 15:23:06.074661 5446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:34103 every 8 connection(s)
I20260924 15:23:06.074967 5317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[4939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263386, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:06.083377 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 5317
I20260924 15:23:06.083478 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:06.083711 4409 external_mini_cluster.cc:1428] Setting key ccf592887e74179282487756db140143
I20260924 15:23:06.084410 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.075793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:53467 (local address 127.4.78.126:45463)
0924 15:23:06.075876 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.075878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.076025 (+ 147us) server_negotiation.cc:407] Connection header received
0924 15:23:06.077035 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.077036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.077048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.077114 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:06.077709 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.078272 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.079298 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.079489 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.079550 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.081598 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:06.081599 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:06.081600 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:06.081605 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:06.082903 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.083279 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.083280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.083280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.083347 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.083667 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.083668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.083669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.083811 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:06.083891 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.084218 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.084313 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 15:23:06.084622 5449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.075703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45463 (local address 127.4.78.67:53467)
0924 15:23:06.075890 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:06.075933 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:06.076933 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:06.077191 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:06.077195 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:06.077442 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:06.077633 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.077637 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.078398 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.078399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.079205 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.079207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.079565 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.079566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.079664 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.080211 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:06.080219 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:06.081513 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:06.082979 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:06.082985 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:06.082986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:06.083200 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:06.083421 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:06.083421 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:06.083422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:06.083601 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:06.083877 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:06.083884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:06.084039 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 15:23:06.084156 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 15:23:06.084379 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":99,"threads_started":1}
I20260924 15:23:06.085033 5447 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45463
I20260924 15:23:06.085141 5447 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:06.085322 5447 heartbeater.cc:511] Master 127.4.78.126:45463 requested a full tablet report, sending...
I20260924 15:23:06.085716 4985 ts_manager.cc:194] Registered new tserver with Master: 6821e8fc96d94111838ac7a0b0f4136e (127.4.78.67:34103)
I20260924 15:23:06.086118 4985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:53467
I20260924 15:23:06.095520 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:23:06.103562 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.097133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41776 (local address 127.4.78.126:45463)
0924 15:23:06.097211 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.097213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.097301 (+ 88us) server_negotiation.cc:407] Connection header received
0924 15:23:06.097381 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.097381 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.097392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.097449 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:06.098099 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.098602 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.099332 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.099532 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.099631 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.100878 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:06.100879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:06.100880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:06.100887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:06.102334 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.102729 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.102731 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.102731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.102794 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.103001 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.103002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.103002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.103155 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:06.103268 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.103376 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.103457 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:23:06.105943 4985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:06.107545 4985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:06.114082 5459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.109925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:34103 (local address 127.0.0.1:47216)
0924 15:23:06.110200 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:06.110209 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:06.110273 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:06.110707 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:06.110708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:06.110710 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:06.110769 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.110770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.111706 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.111707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.112946 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.112949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.113720 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.113721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.113803 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.113847 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 15:23:06.113942 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 15:23:06.113999 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":5,"thread_start_us":83,"threads_started":1}
I20260924 15:23:06.114255 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.110102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43405 (local address 127.0.0.1:41876)
0924 15:23:06.110326 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:06.110339 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:06.110396 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:06.110645 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:06.110648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:06.110651 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:06.110723 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.110724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.112035 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.112036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.113061 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.113062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.113914 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.113915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.113972 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.113976 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:06.114021 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 15:23:06.114053 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260924 15:23:06.114354 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.109925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:37125 (local address 127.0.0.1:41816)
0924 15:23:06.110157 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:06.110202 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:06.110273 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:06.111012 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:06.111013 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:06.111015 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:06.111061 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.111063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.112274 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.112275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.113219 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.113221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.114160 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.114161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.114218 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.114222 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:06.114249 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 15:23:06.114286 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":44,"threads_started":1}
I20260924 15:23:06.114385 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.110318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41876 (local address 127.4.78.66:43405)
0924 15:23:06.110483 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.110487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.110496 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:06.110527 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.110528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.110540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.110607 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:06.110815 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.111920 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.113139 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.113810 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.113906 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.113960 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.114046 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.114103 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260924 15:23:06.114451 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.110016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47216 (local address 127.4.78.67:34103)
0924 15:23:06.110521 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.110525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.110533 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:06.110564 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.110565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.110578 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.110687 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:06.110856 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.111600 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.113029 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.113634 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.113721 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.113791 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.114299 (+ 508us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.114354 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":99,"threads_started":1}
I20260924 15:23:06.115064 5245 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8f060ccba74e1aa3319a1ea533f904 (DEFAULT_TABLE table=test-table [id=8f399fef514e4bc7bc1701c2a166508f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:06.115147 5380 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8f060ccba74e1aa3319a1ea533f904 (DEFAULT_TABLE table=test-table [id=8f399fef514e4bc7bc1701c2a166508f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:06.115453 5245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8f060ccba74e1aa3319a1ea533f904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:06.115489 5380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8f060ccba74e1aa3319a1ea533f904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:06.117863 5464 tablet_bootstrap.cc:492] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e: Bootstrap starting.
I20260924 15:23:06.117924 5465 tablet_bootstrap.cc:492] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99: Bootstrap starting.
I20260924 15:23:06.118366 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.110112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41816 (local address 127.4.78.65:37125)
0924 15:23:06.110846 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.110851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.110861 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:06.110897 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.110899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.110913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.111132 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:06.111210 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.112158 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.113321 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.114067 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.114454 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.118178 (+ 3724us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.118207 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.118262 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":81,"threads_started":1}
I20260924 15:23:06.118673 5464 tablet_bootstrap.cc:654] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:06.118669 5465 tablet_bootstrap.cc:654] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:06.119005 5110 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8f060ccba74e1aa3319a1ea533f904 (DEFAULT_TABLE table=test-table [id=8f399fef514e4bc7bc1701c2a166508f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:06.119225 5465 log.cc:824] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:06.119234 5464 log.cc:824] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:06.119340 5110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a8f060ccba74e1aa3319a1ea533f904. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:06.119976 5464 tablet_bootstrap.cc:492] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e: No bootstrap required, opened a new log
I20260924 15:23:06.120082 5464 ts_tablet_manager.cc:1397] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:06.120236 5465 tablet_bootstrap.cc:492] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99: No bootstrap required, opened a new log
I20260924 15:23:06.120318 5465 ts_tablet_manager.cc:1397] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:23:06.121544 5464 raft_consensus.cc:348] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.121738 5464 raft_consensus.cc:374] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:06.121785 5464 raft_consensus.cc:729] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6821e8fc96d94111838ac7a0b0f4136e, State: Initialized, Role: FOLLOWER
I20260924 15:23:06.121922 5464 consensus_queue.cc:260] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.122222 5447 heartbeater.cc:503] Master 127.4.78.126:45463 was elected leader, sending a full tablet report...
I20260924 15:23:06.122414 5465 raft_consensus.cc:348] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.122583 5465 raft_consensus.cc:374] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:06.122226 5464 ts_tablet_manager.cc:1428] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:06.122627 5465 raft_consensus.cc:729] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8048a0d417a47bf92bd7ef451d73f99, State: Initialized, Role: FOLLOWER
I20260924 15:23:06.122715 5469 tablet_bootstrap.cc:492] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3: Bootstrap starting.
I20260924 15:23:06.122728 5465 consensus_queue.cc:260] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.123003 5465 ts_tablet_manager.cc:1428] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:23:06.123116 5312 heartbeater.cc:503] Master 127.4.78.126:45463 was elected leader, sending a full tablet report...
I20260924 15:23:06.123342 5469 tablet_bootstrap.cc:654] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:06.123907 5469 log.cc:824] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:06.124567 5469 tablet_bootstrap.cc:492] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3: No bootstrap required, opened a new log
I20260924 15:23:06.124646 5469 ts_tablet_manager.cc:1397] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:06.126240 5469 raft_consensus.cc:348] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.126386 5469 raft_consensus.cc:374] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:06.126431 5469 raft_consensus.cc:729] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6577de42ffe84a15a710976a9fd9aca3, State: Initialized, Role: FOLLOWER
I20260924 15:23:06.126539 5469 consensus_queue.cc:260] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.126796 5469 ts_tablet_manager.cc:1428] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:06.126859 5177 heartbeater.cc:503] Master 127.4.78.126:45463 was elected leader, sending a full tablet report...
W20260924 15:23:06.163987 5313 tablet.cc:2369] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:23:06.246701 5178 tablet.cc:2369] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:23:06.325702 5448 tablet.cc:2369] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:06.354501 5472 raft_consensus.cc:482] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:06.354625 5472 raft_consensus.cc:504] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.355013 5472 leader_election.cc:288] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8048a0d417a47bf92bd7ef451d73f99 (127.4.78.66:43405), 6821e8fc96d94111838ac7a0b0f4136e (127.4.78.67:34103)
I20260924 15:23:06.357995 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.355291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:40547 (local address 127.4.78.66:43405)
0924 15:23:06.355338 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.355340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.355460 (+ 120us) server_negotiation.cc:407] Connection header received
0924 15:23:06.355550 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.355551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.355570 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.355637 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:06.355802 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.356304 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.357247 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.357615 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.357700 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.357709 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.357881 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.357919 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:23:06.358021 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.355228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43405 (local address 127.4.78.65:40547)
0924 15:23:06.355376 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:06.355386 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:06.355471 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:06.355642 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:06.355643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:06.355646 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:06.355731 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.355732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.356396 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.356397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.357150 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.357152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.357703 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.357704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.357788 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.357791 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:06.357876 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:23:06.357919 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260924 15:23:06.358031 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.355331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:50069 (local address 127.4.78.67:34103)
0924 15:23:06.355376 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.355378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.355426 (+ 48us) server_negotiation.cc:407] Connection header received
0924 15:23:06.355515 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.355515 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.355534 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.355601 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:06.355766 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.356253 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.357218 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.357609 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.357697 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.357707 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.357916 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.357950 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:23:06.358150 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.355214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:34103 (local address 127.4.78.65:50069)
0924 15:23:06.355359 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:06.355372 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:06.355440 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:06.355606 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:06.355607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:06.355610 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:06.355685 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.355687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.356342 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.356343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.357120 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:06.357123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.357701 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.357702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:06.357781 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.357785 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:06.357901 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 15:23:06.357944 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260924 15:23:06.358579 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a8f060ccba74e1aa3319a1ea533f904" candidate_uuid: "6577de42ffe84a15a710976a9fd9aca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6821e8fc96d94111838ac7a0b0f4136e" is_pre_election: true
I20260924 15:23:06.358583 5266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a8f060ccba74e1aa3319a1ea533f904" candidate_uuid: "6577de42ffe84a15a710976a9fd9aca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8048a0d417a47bf92bd7ef451d73f99" is_pre_election: true
I20260924 15:23:06.358738 5400 raft_consensus.cc:2454] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6577de42ffe84a15a710976a9fd9aca3 in term 0.
I20260924 15:23:06.358763 5266 raft_consensus.cc:2454] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6577de42ffe84a15a710976a9fd9aca3 in term 0.
I20260924 15:23:06.359012 5063 leader_election.cc:302] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6577de42ffe84a15a710976a9fd9aca3, c8048a0d417a47bf92bd7ef451d73f99; no voters:
I20260924 15:23:06.359180 5472 raft_consensus.cc:2781] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:06.359246 5472 raft_consensus.cc:482] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:06.359263 5472 raft_consensus.cc:3037] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:06.359964 5472 raft_consensus.cc:504] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.360110 5472 leader_election.cc:288] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [CANDIDATE]: Term 1 election: Requested vote from peers c8048a0d417a47bf92bd7ef451d73f99 (127.4.78.66:43405), 6821e8fc96d94111838ac7a0b0f4136e (127.4.78.67:34103)
I20260924 15:23:06.360234 5266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a8f060ccba74e1aa3319a1ea533f904" candidate_uuid: "6577de42ffe84a15a710976a9fd9aca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8048a0d417a47bf92bd7ef451d73f99"
I20260924 15:23:06.360293 5400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a8f060ccba74e1aa3319a1ea533f904" candidate_uuid: "6577de42ffe84a15a710976a9fd9aca3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6821e8fc96d94111838ac7a0b0f4136e"
I20260924 15:23:06.360308 5266 raft_consensus.cc:3037] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:06.360363 5400 raft_consensus.cc:3037] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:06.361213 5400 raft_consensus.cc:2454] T 2a8f060ccba74e1aa3319a1ea533f904 P 6821e8fc96d94111838ac7a0b0f4136e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6577de42ffe84a15a710976a9fd9aca3 in term 1.
I20260924 15:23:06.361213 5266 raft_consensus.cc:2454] T 2a8f060ccba74e1aa3319a1ea533f904 P c8048a0d417a47bf92bd7ef451d73f99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6577de42ffe84a15a710976a9fd9aca3 in term 1.
I20260924 15:23:06.361382 5063 leader_election.cc:302] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6577de42ffe84a15a710976a9fd9aca3, c8048a0d417a47bf92bd7ef451d73f99; no voters:
I20260924 15:23:06.361485 5472 raft_consensus.cc:2781] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:06.361629 5472 raft_consensus.cc:686] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [term 1 LEADER]: Becoming Leader. State: Replica: 6577de42ffe84a15a710976a9fd9aca3, State: Running, Role: LEADER
I20260924 15:23:06.361733 5472 consensus_queue.cc:237] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } }
I20260924 15:23:06.362694 4984 catalog_manager.cc:5495] T 2a8f060ccba74e1aa3319a1ea533f904 P 6577de42ffe84a15a710976a9fd9aca3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6577de42ffe84a15a710976a9fd9aca3 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "6577de42ffe84a15a710976a9fd9aca3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6577de42ffe84a15a710976a9fd9aca3" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8048a0d417a47bf92bd7ef451d73f99" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6821e8fc96d94111838ac7a0b0f4136e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34103 } health_report { overall_health: UNKNOWN } } }
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[4939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:06.402781 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.394219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.4.78.65:37125)
0924 15:23:06.394288 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.394290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.394297 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:06.394320 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.394321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.394333 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.394408 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:06.395016 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.395506 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.396308 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.396523 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.396609 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.399367 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:06.399386 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:06.399386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:06.399393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:06.401493 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.401861 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.401866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.401866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.401948 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.402161 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.402162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.402163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.402361 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:06.402511 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.402555 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.402654 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[4939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263386, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:06.425803 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.418577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.4.78.65:37125)
0924 15:23:06.418655 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.418657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.418736 (+ 79us) server_negotiation.cc:407] Connection header received
0924 15:23:06.418808 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.418808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.418820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.418891 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:06.419432 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.419943 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.420670 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.420868 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.420951 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.423209 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:06.423210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:06.423210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:06.423224 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:06.424606 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.424971 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.424972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.424972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.425028 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.425227 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.425228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.425229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.425328 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:06.425458 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.425620 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.425701 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[4939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[4939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:06.448776 5462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:06.441409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.4.78.65:37125)
0924 15:23:06.441498 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:06.441500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:06.441590 (+ 90us) server_negotiation.cc:407] Connection header received
0924 15:23:06.441686 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:06.441687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:06.441697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:06.441773 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:06.442355 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.442830 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.443572 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:06.443775 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:06.443849 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:06.446423 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:06.446425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:06.446425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:06.446431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:06.447728 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.448118 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.448118 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.448119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.448171 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:06.448364 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:06.448364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:06.448364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:06.448447 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:06.448520 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:06.448659 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 15:23:06.448713 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260924 15:23:06.449352 5090 tablet_service.cc:2908] Rejecting scan request for tablet 2a8f060ccba74e1aa3319a1ea533f904: Uninitialized: clean time has not yet been initialized
I20260924 15:23:06.450241 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 5047
I20260924 15:23:06.455946 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 5182
I20260924 15:23:06.461644 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 5317
I20260924 15:23:06.466279 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 4952
2026-09-24T15:23:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2229 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[5489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[5489](info): set up 2 sockets
Sep 24 15:23:06 dist-test-slave-l766 krb5kdc[5489](info): commencing operation
krb5kdc: starting...
W20260924 15:23:07.508471 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:07 dist-test-slave-l766 krb5kdc[5489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:07.621670 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:33207
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:33207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:07.702476 5502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:07.702677 5502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:07.702704 5502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:07.703894 5502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:07.703943 5502 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:23:07.703958 5502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:07.703970 5502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:07.703984 5502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:07.703995 5502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:07.705457 5502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:33207
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:33207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:07.705703 5502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:07.705960 5502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:07.710831 5508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:07.710891 5502 server_base.cc:1034] running on GCE node
W20260924 15:23:07.710829 5510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:07.710831 5507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:07.711309 5502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:07.711628 5502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:07.712800 5502 hybrid_clock.cc:648] HybridClock initialized: now 1790263387712783 us; error 26 us; skew 500 ppm
Sep 24 15:23:07 dist-test-slave-l766 krb5kdc[5489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:07.715430 5502 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:07.715930 5502 webserver.cc:460] Webserver started at http://127.4.78.126:33031/ using document root <none> and password file <none>
I20260924 15:23:07.716171 5502 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:07.716239 5502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:07.716396 5502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:07.717427 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "2d29649bf8554083b3bd840ff88b4fe2"
format_stamp: "Formatted at 2026-09-24 15:23:07 on dist-test-slave-l766"
server_key: "3c12cd504938241e29e7ce47fef25186"
server_key_iv: "23410c94f1c2abcd6dcf07fd8d5af379"
server_key_version: "encryptionkey@0"
I20260924 15:23:07.717801 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "2d29649bf8554083b3bd840ff88b4fe2"
format_stamp: "Formatted at 2026-09-24 15:23:07 on dist-test-slave-l766"
server_key: "3c12cd504938241e29e7ce47fef25186"
server_key_iv: "23410c94f1c2abcd6dcf07fd8d5af379"
server_key_version: "encryptionkey@0"
I20260924 15:23:07.719237 5502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:07.720250 5517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:07.720439 5502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:07.720547 5502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "2d29649bf8554083b3bd840ff88b4fe2"
format_stamp: "Formatted at 2026-09-24 15:23:07 on dist-test-slave-l766"
server_key: "3c12cd504938241e29e7ce47fef25186"
server_key_iv: "23410c94f1c2abcd6dcf07fd8d5af379"
server_key_version: "encryptionkey@0"
I20260924 15:23:07.720629 5502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:07.737318 5502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:07.739969 5502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:07.740150 5502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:07.745257 5502 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:33207
I20260924 15:23:07.745389 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:33207 every 8 connection(s)
I20260924 15:23:07.745692 5502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:07.746282 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 5502
I20260924 15:23:07.746400 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:07.746502 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:07.746585 4409 external_mini_cluster.cc:1428] Setting key 1638e77a63120e3403cde46dd4d87bac
I20260924 15:23:07.750300 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2: Bootstrap starting.
I20260924 15:23:07.750983 5580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:07.751288 5580 log.cc:824] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:07.752038 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2: No bootstrap required, opened a new log
Sep 24 15:23:07 dist-test-slave-l766 krb5kdc[5489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:07.754139 5580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d29649bf8554083b3bd840ff88b4fe2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33207 } }
I20260924 15:23:07.754325 5580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:07.754390 5580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d29649bf8554083b3bd840ff88b4fe2, State: Initialized, Role: FOLLOWER
I20260924 15:23:07.754581 5580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d29649bf8554083b3bd840ff88b4fe2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33207 } }
I20260924 15:23:07.754683 5580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:07.754724 5580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:07.754781 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:07.755303 5580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d29649bf8554083b3bd840ff88b4fe2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33207 } }
I20260924 15:23:07.755431 5580 leader_election.cc:302] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d29649bf8554083b3bd840ff88b4fe2; no voters:
I20260924 15:23:07.755625 5580 leader_election.cc:288] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:07.755682 5585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:07.755818 5585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [term 1 LEADER]: Becoming Leader. State: Replica: 2d29649bf8554083b3bd840ff88b4fe2, State: Running, Role: LEADER
I20260924 15:23:07.755898 5580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:07.755968 5585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d29649bf8554083b3bd840ff88b4fe2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33207 } }
I20260924 15:23:07.756351 5585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d29649bf8554083b3bd840ff88b4fe2. Latest consensus state: current_term: 1 leader_uuid: "2d29649bf8554083b3bd840ff88b4fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d29649bf8554083b3bd840ff88b4fe2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33207 } } }
I20260924 15:23:07.756587 5585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:07.756515 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d29649bf8554083b3bd840ff88b4fe2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d29649bf8554083b3bd840ff88b4fe2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33207 } } }
I20260924 15:23:07.756979 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:07.757138 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:07.757642 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:07.758150 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:07.747121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.4.78.126:33207)
0924 15:23:07.747343 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:07.747354 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:07.747389 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:23:07.747917 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:07.747926 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:07.747996 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:07.748175 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:07.748806 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.750219 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.751007 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.751610 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.751734 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:07.754124 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:07.754132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:07.754132 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:07.754142 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:07.756244 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:07.756720 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:07.756723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:07.756724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:07.756833 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:07.757084 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:07.757085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:07.757085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:07.757483 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:07.757644 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:07.757744 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:23:07.757910 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":106,"threads_started":1}
I20260924 15:23:07.759426 5594 catalog_manager.cc:1347] Generated new cluster ID: 48d55920c53b44eaaf585437c5ddb1f5
I20260924 15:23:07.759511 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:07.769474 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:07.770108 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:07.775662 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2: Generated new TSK 0
I20260924 15:23:07.775856 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:07.805111 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:33207
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:07.888051 5606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:07.888268 5606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:07.888331 5606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:07.889642 5606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:07.889681 5606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:07.889703 5606 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:07.889720 5606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:07.891152 5606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:33207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:07.891355 5606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:07.891536 5606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:07.896459 5611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:07.896554 5612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:07.896662 5614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:07.897071 5606 server_base.cc:1034] running on GCE node
I20260924 15:23:07.897300 5606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:07.897528 5606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:07.898691 5606 hybrid_clock.cc:648] HybridClock initialized: now 1790263387898685 us; error 36 us; skew 500 ppm
Sep 24 15:23:07 dist-test-slave-l766 krb5kdc[5489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:07.901147 5606 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:07.901660 5606 webserver.cc:460] Webserver started at http://127.4.78.65:46715/ using document root <none> and password file <none>
I20260924 15:23:07.901973 5606 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:07.902045 5606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:07.902199 5606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:07.903263 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a"
format_stamp: "Formatted at 2026-09-24 15:23:07 on dist-test-slave-l766"
server_key: "cbd792d43e7c09eef1d5a250b8d193ce"
server_key_iv: "3ebf223e55bc84b7d0199a2b6a165de4"
server_key_version: "encryptionkey@0"
I20260924 15:23:07.903661 5606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a"
format_stamp: "Formatted at 2026-09-24 15:23:07 on dist-test-slave-l766"
server_key: "cbd792d43e7c09eef1d5a250b8d193ce"
server_key_iv: "3ebf223e55bc84b7d0199a2b6a165de4"
server_key_version: "encryptionkey@0"
I20260924 15:23:07.904980 5606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:07.905856 5621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:07.906051 5606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:07.906152 5606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a"
format_stamp: "Formatted at 2026-09-24 15:23:07 on dist-test-slave-l766"
server_key: "cbd792d43e7c09eef1d5a250b8d193ce"
server_key_iv: "3ebf223e55bc84b7d0199a2b6a165de4"
server_key_version: "encryptionkey@0"
I20260924 15:23:07.906240 5606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:07.943216 5606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:07.945145 5606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:07.945322 5606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:07.945926 5606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:07.945990 5606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:07.946033 5606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:07.946060 5606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:23:07 dist-test-slave-l766 krb5kdc[5489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:07.953537 5606 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:42585
I20260924 15:23:07.953922 5737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:42585 every 8 connection(s)
I20260924 15:23:07.953977 5606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:07.956261 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:07.946414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:55765 (local address 127.4.78.126:33207)
0924 15:23:07.946462 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:07.946464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:07.946684 (+ 220us) server_negotiation.cc:407] Connection header received
0924 15:23:07.947279 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:07.947281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:07.947293 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:07.947351 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:07.948142 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.948740 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.949847 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.950112 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.950187 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:07.953302 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:07.953304 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:07.953304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:07.953313 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:07.954950 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:07.955359 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:07.955360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:07.955360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:07.955413 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:07.955635 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:07.955636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:07.955636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:07.955772 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:07.955836 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:07.956124 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 15:23:07.956196 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:23:07.956432 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:07.946405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:33207 (local address 127.4.78.65:55765)
0924 15:23:07.946563 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:07.946599 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:07.947171 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:07.947424 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:07.947427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:07.947743 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:07.947927 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:07.947931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.948834 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.948835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:07.949727 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:07.949730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.950206 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.950207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:07.950337 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:07.950867 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:07.950879 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:07.953208 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:07.955050 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:07.955058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:07.955060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:07.955282 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:07.955472 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:07.955473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:07.955473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:07.955579 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:07.955869 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:07.955880 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:07.956000 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 15:23:07.956084 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 15:23:07.956204 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260924 15:23:07.956980 5738 heartbeater.cc:347] Connected to a master server at 127.4.78.126:33207
I20260924 15:23:07.957070 5738 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:07.957270 5738 heartbeater.cc:511] Master 127.4.78.126:33207 requested a full tablet report, sending...
I20260924 15:23:07.957913 5534 ts_manager.cc:194] Registered new tserver with Master: 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585)
I20260924 15:23:07.958570 5534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:55765
I20260924 15:23:07.960151 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 5606
I20260924 15:23:07.960264 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:07.960438 4409 external_mini_cluster.cc:1428] Setting key e1fdb8fe145623c4dbff887a92fbb9e4
I20260924 15:23:07.964484 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:07.958138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:57937 (local address 127.4.78.126:33207)
0924 15:23:07.958175 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:07.958177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:07.958294 (+ 117us) server_negotiation.cc:407] Connection header received
0924 15:23:07.958366 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:07.958367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:07.958376 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:07.958421 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:07.958751 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.959399 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.960114 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.960402 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.960473 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:07.961395 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:07.961397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:07.961397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:07.961403 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:07.963318 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:07.963659 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:07.963660 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:07.963660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:07.963707 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:07.963913 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:07.963914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:07.963914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:07.964027 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:07.964168 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:07.964319 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 15:23:07.964394 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:23:07.964490 5744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:07.958096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:33207 (local address 127.4.78.65:57937)
0924 15:23:07.958221 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:07.958229 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:07.958308 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:07.958433 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:07.958434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:07.958610 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:07.958692 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:07.958694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.959486 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.959486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:07.960009 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:07.960010 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:07.960471 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:07.960472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:07.960548 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:07.960998 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:07.961003 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:07.961290 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:07.963403 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:07.963404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:07.963404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:07.963589 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:07.963784 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:07.963785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:07.963785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:07.963849 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:07.964162 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:07.964163 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:07.964210 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 15:23:07.964293 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:23:07.964397 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:07.988965 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:33207
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:08.071461 5747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:08.071658 5747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:08.071682 5747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:08.072850 5747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:08.072893 5747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:08.072913 5747 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:08.072968 5747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:08.074450 5747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:33207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:08.074724 5747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:08.074954 5747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:08.079871 5752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:08.079872 5755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:08.079877 5753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:08.079888 5747 server_base.cc:1034] running on GCE node
I20260924 15:23:08.080240 5747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:08.080467 5747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:08.081640 5747 hybrid_clock.cc:648] HybridClock initialized: now 1790263388081626 us; error 24 us; skew 500 ppm
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[5489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:08.084153 5747 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:08.084523 5747 webserver.cc:460] Webserver started at http://127.4.78.66:39929/ using document root <none> and password file <none>
I20260924 15:23:08.084700 5747 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:08.084740 5747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:08.084826 5747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:08.085778 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "c0d7db3566444619a3f5eda57c6480a0"
format_stamp: "Formatted at 2026-09-24 15:23:08 on dist-test-slave-l766"
server_key: "d97d05e8a8673a10f7a231dc6ecb94bb"
server_key_iv: "cc10451a1441c508fcf1e446f05383b2"
server_key_version: "encryptionkey@0"
I20260924 15:23:08.086155 5747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "c0d7db3566444619a3f5eda57c6480a0"
format_stamp: "Formatted at 2026-09-24 15:23:08 on dist-test-slave-l766"
server_key: "d97d05e8a8673a10f7a231dc6ecb94bb"
server_key_iv: "cc10451a1441c508fcf1e446f05383b2"
server_key_version: "encryptionkey@0"
I20260924 15:23:08.087368 5747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:08.088084 5762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.088315 5747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:08.088402 5747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "c0d7db3566444619a3f5eda57c6480a0"
format_stamp: "Formatted at 2026-09-24 15:23:08 on dist-test-slave-l766"
server_key: "d97d05e8a8673a10f7a231dc6ecb94bb"
server_key_iv: "cc10451a1441c508fcf1e446f05383b2"
server_key_version: "encryptionkey@0"
I20260924 15:23:08.088455 5747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:08.118237 5747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:08.120169 5747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:08.120292 5747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:08.120759 5747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:08.120792 5747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.120815 5747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:08.120833 5747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.128126 5747 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:39527
I20260924 15:23:08.128182 5877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:39527 every 8 connection(s)
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[5489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:08.128486 5747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:08.131382 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.121355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:41615 (local address 127.4.78.126:33207)
0924 15:23:08.121413 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.121415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.121651 (+ 236us) server_negotiation.cc:407] Connection header received
0924 15:23:08.122276 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.122278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.122293 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.122370 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:08.123306 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.124137 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.125985 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.126299 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.126358 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.129040 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:08.129042 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:08.129042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:08.129050 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:08.130261 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.130642 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.130643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.130643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.130693 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.130907 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.130907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.130907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.130999 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:08.131049 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.131258 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.131321 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:23:08.131605 5771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.121340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:33207 (local address 127.4.78.66:41615)
0924 15:23:08.121520 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.121561 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.122133 (+ 572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.122464 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.122468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.122864 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:08.123204 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.123213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.124264 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.124266 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.125886 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.125889 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.126385 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.126386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.126556 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.127104 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:08.127115 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:08.128929 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:08.130329 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.130334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.130336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.130575 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.130756 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.130757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.130757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.130851 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.131052 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:08.131055 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:08.131159 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.131248 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.131360 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":102,"threads_started":1}
I20260924 15:23:08.132090 5878 heartbeater.cc:347] Connected to a master server at 127.4.78.126:33207
I20260924 15:23:08.132167 5878 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:08.132345 5878 heartbeater.cc:511] Master 127.4.78.126:33207 requested a full tablet report, sending...
I20260924 15:23:08.132752 5533 ts_manager.cc:194] Registered new tserver with Master: c0d7db3566444619a3f5eda57c6480a0 (127.4.78.66:39527)
I20260924 15:23:08.133078 5533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:41615
I20260924 15:23:08.135375 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 5747
I20260924 15:23:08.135497 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:08.135701 4409 external_mini_cluster.cc:1428] Setting key f3572fc2824d103add881bf644e1be91
I20260924 15:23:08.138648 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.133185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:51657 (local address 127.4.78.126:33207)
0924 15:23:08.133222 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.133224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.133401 (+ 177us) server_negotiation.cc:407] Connection header received
0924 15:23:08.133450 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.133451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.133459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.133504 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:08.133857 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.134302 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.134939 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.135107 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.135154 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.136163 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:08.136164 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:08.136164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:08.136171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:08.137491 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.137856 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.137857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.137857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.137910 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.138112 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.138112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.138113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.138263 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:08.138339 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.138485 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.138557 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:23:08.138654 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.133192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:33207 (local address 127.4.78.66:51657)
0924 15:23:08.133327 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.133337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.133398 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.133508 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.133509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.133698 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:08.133754 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.133756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.134385 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.134386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.134875 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.134877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.135190 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.135191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.135276 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.135724 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:08.135729 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:08.136091 (+ 362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:08.137577 (+ 1486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.137577 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.137578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.137758 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.137980 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.137981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.137981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.138038 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.138332 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:08.138333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:08.138378 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.138460 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.138558 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:08.160379 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:33207
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:08.253798 5887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:08.254138 5887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:08.254185 5887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:08.256507 5887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:08.256567 5887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:08.256605 5887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:08.256639 5887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:08.259403 5887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:33207
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:08.259685 5887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:08.259963 5887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:08.265571 5892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:08.265594 5895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:08.265949 5893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:08.266894 5887 server_base.cc:1034] running on GCE node
I20260924 15:23:08.267107 5887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:08.267349 5887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:08.268534 5887 hybrid_clock.cc:648] HybridClock initialized: now 1790263388268510 us; error 42 us; skew 500 ppm
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[5489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:08.271170 5887 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:08.271597 5887 webserver.cc:460] Webserver started at http://127.4.78.67:33225/ using document root <none> and password file <none>
I20260924 15:23:08.271842 5887 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:08.271912 5887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:08.272069 5887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:08.273150 5887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "e7c56c3a54154abbb183d4a39c7406fc"
format_stamp: "Formatted at 2026-09-24 15:23:08 on dist-test-slave-l766"
server_key: "31086b442f42e8c7b1a936c304f8a23c"
server_key_iv: "14f68ccd9fa79f4653ec53731ea89321"
server_key_version: "encryptionkey@0"
I20260924 15:23:08.273507 5887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "e7c56c3a54154abbb183d4a39c7406fc"
format_stamp: "Formatted at 2026-09-24 15:23:08 on dist-test-slave-l766"
server_key: "31086b442f42e8c7b1a936c304f8a23c"
server_key_iv: "14f68ccd9fa79f4653ec53731ea89321"
server_key_version: "encryptionkey@0"
I20260924 15:23:08.274858 5887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:08.275673 5902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.275872 5887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:08.275949 5887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "e7c56c3a54154abbb183d4a39c7406fc"
format_stamp: "Formatted at 2026-09-24 15:23:08 on dist-test-slave-l766"
server_key: "31086b442f42e8c7b1a936c304f8a23c"
server_key_iv: "14f68ccd9fa79f4653ec53731ea89321"
server_key_version: "encryptionkey@0"
I20260924 15:23:08.276001 5887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:08.298192 5887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:08.300297 5887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:08.300478 5887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:08.301004 5887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:08.301076 5887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.301113 5887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:08.301158 5887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[5489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263388, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:08.308532 5887 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:40399
I20260924 15:23:08.308640 6017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:40399 every 8 connection(s)
I20260924 15:23:08.308974 5887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:08.311653 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.301492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:38205 (local address 127.4.78.126:33207)
0924 15:23:08.301538 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.301540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.301782 (+ 242us) server_negotiation.cc:407] Connection header received
0924 15:23:08.302363 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.302365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.302379 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.302444 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:08.303247 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.304077 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.305744 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.306087 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.306161 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.308664 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:08.308666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:08.308666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:08.308674 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:08.310274 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.310691 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.310693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.310693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.310754 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.311009 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.311010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.311011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.311136 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:08.311212 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.311481 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.311562 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:23:08.311834 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.301477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:33207 (local address 127.4.78.67:38205)
0924 15:23:08.301660 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.301693 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.302255 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.302513 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.302517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.302865 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:08.303162 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.303169 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.304185 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.304187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.305658 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.305661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.306200 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.306202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.306340 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.306880 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:08.306888 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:08.308555 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:08.310375 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.310382 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.310385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.310605 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.310822 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.310823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.310823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.310934 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.311203 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:08.311209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:08.311345 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.311449 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.311613 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260924 15:23:08.312351 6018 heartbeater.cc:347] Connected to a master server at 127.4.78.126:33207
I20260924 15:23:08.312431 6018 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:08.312616 6018 heartbeater.cc:511] Master 127.4.78.126:33207 requested a full tablet report, sending...
I20260924 15:23:08.313031 5534 ts_manager.cc:194] Registered new tserver with Master: e7c56c3a54154abbb183d4a39c7406fc (127.4.78.67:40399)
I20260924 15:23:08.313508 5534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:38205
I20260924 15:23:08.315419 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 5887
I20260924 15:23:08.315510 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:08.315680 4409 external_mini_cluster.cc:1428] Setting key 1b22416e0568c2ed9b831ce92ed28816
I20260924 15:23:08.316396 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:23:08.319379 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.313393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:55899 (local address 127.4.78.126:33207)
0924 15:23:08.313634 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.313635 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.313643 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:08.313663 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.313663 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.313672 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.313722 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:08.314120 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.314651 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.315431 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.315712 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.315793 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.316751 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:08.316753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:08.316753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:08.316759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:08.318340 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.318736 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.318737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.318737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.318783 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.318963 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.318963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.318964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.319042 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:08.319091 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.319278 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.319328 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260924 15:23:08.319490 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.313334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:33207 (local address 127.4.78.67:55899)
0924 15:23:08.313445 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.313455 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.313506 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.313740 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.313741 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.313956 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:08.314038 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.314040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.314757 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.314758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.315314 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.315315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.315801 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.315802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.315876 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.316312 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:08.316318 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:08.316651 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:08.318436 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.318437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.318438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.318666 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.318842 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.318843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.318843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.318909 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.319129 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:08.319130 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:08.319181 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.319262 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.319368 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[5489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[5489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:08.339732 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.332645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50734 (local address 127.4.78.126:33207)
0924 15:23:08.332705 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.332707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.332718 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:23:08.332798 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.332799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.332809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.332885 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:08.333492 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.334096 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.334867 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.335052 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.335143 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.337333 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:08.337334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:08.337335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:08.337343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:08.338622 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.338965 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.338966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.338966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.339020 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.339224 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.339225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.339225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.339332 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:08.339441 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.339543 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.339627 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:23:08.341199 5534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:08.342759 5534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:08.349269 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.344993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:42585 (local address 127.0.0.1:46000)
0924 15:23:08.345221 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.345505 (+ 284us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.345568 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.345900 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.345901 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.345904 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:08.345952 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.345954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.346831 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.346832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.348086 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.348098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.349019 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.349020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.349100 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.349103 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.349155 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.349187 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":45,"threads_started":1}
I20260924 15:23:08.349352 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.345093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.4.78.65:42585)
0924 15:23:08.345323 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.345328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.345561 (+ 233us) server_negotiation.cc:407] Connection header received
0924 15:23:08.345636 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.345639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.345655 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.345838 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:08.346031 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.346733 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.348189 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.348932 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.348999 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.349102 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.349167 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.349247 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":120,"threads_started":1}
I20260924 15:23:08.349375 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.345130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38804 (local address 127.4.78.66:39527)
0924 15:23:08.345324 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.345328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.345466 (+ 138us) server_negotiation.cc:407] Connection header received
0924 15:23:08.345590 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.345592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.345616 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.345699 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:08.345871 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.347015 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.348242 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.348759 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.348839 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.348897 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.349231 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.349274 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260924 15:23:08.349189 6036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.345030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:39527 (local address 127.0.0.1:38804)
0924 15:23:08.345359 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.345396 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.345481 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.345703 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.345705 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.345708 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:08.345770 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.345772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.347102 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.347102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.348156 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.348158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.348839 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.348839 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.348916 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.348929 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.348999 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.349100 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
I20260924 15:23:08.349736 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.345244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40399 (local address 127.0.0.1:34736)
0924 15:23:08.345653 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.345662 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.345705 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.345987 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.345988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.345989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:08.346028 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.346029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.347234 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.347235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.348115 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.348117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.349456 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.349457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.349541 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.349545 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.349631 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.349669 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260924 15:23:08.349998 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.345341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34736 (local address 127.4.78.67:40399)
0924 15:23:08.345492 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.345496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.345762 (+ 266us) server_negotiation.cc:407] Connection header received
0924 15:23:08.345802 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.345804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.345861 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.345926 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:08.346108 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.347157 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.348224 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.348932 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.349002 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.349095 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.349862 (+ 767us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.349904 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260924 15:23:08.350360 5671 tablet_service.cc:1467] Processing CreateTablet for tablet 441214bf4a4b4bbfb7c75d50e4408421 (DEFAULT_TABLE table=test-table [id=0770c67c13c8443589bba1695838effc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:08.350436 5812 tablet_service.cc:1467] Processing CreateTablet for tablet 441214bf4a4b4bbfb7c75d50e4408421 (DEFAULT_TABLE table=test-table [id=0770c67c13c8443589bba1695838effc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:08.350490 5952 tablet_service.cc:1467] Processing CreateTablet for tablet 441214bf4a4b4bbfb7c75d50e4408421 (DEFAULT_TABLE table=test-table [id=0770c67c13c8443589bba1695838effc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:08.350734 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441214bf4a4b4bbfb7c75d50e4408421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.350785 5952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441214bf4a4b4bbfb7c75d50e4408421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.350821 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441214bf4a4b4bbfb7c75d50e4408421. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.353503 6042 tablet_bootstrap.cc:492] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Bootstrap starting.
I20260924 15:23:08.353998 6043 tablet_bootstrap.cc:492] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: Bootstrap starting.
I20260924 15:23:08.354226 6044 tablet_bootstrap.cc:492] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Bootstrap starting.
I20260924 15:23:08.354347 6042 tablet_bootstrap.cc:654] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.354743 6042 log.cc:824] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:08.354777 6044 tablet_bootstrap.cc:654] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.355049 6043 tablet_bootstrap.cc:654] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.355103 6044 log.cc:824] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:08.355396 6043 log.cc:824] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:08.355551 6042 tablet_bootstrap.cc:492] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: No bootstrap required, opened a new log
I20260924 15:23:08.355626 6042 ts_tablet_manager.cc:1397] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:23:08.355854 6044 tablet_bootstrap.cc:492] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: No bootstrap required, opened a new log
I20260924 15:23:08.355924 6044 ts_tablet_manager.cc:1397] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:08.356119 6043 tablet_bootstrap.cc:492] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: No bootstrap required, opened a new log
I20260924 15:23:08.356190 6043 ts_tablet_manager.cc:1397] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:08.357689 6044 raft_consensus.cc:348] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.357869 6044 raft_consensus.cc:374] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.357913 6044 raft_consensus.cc:729] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519f9d6b7ecc4ccaa7a6dce60f065c5a, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.357910 6042 raft_consensus.cc:348] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.358034 6044 consensus_queue.cc:260] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.358108 6042 raft_consensus.cc:374] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.358155 6042 raft_consensus.cc:729] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d7db3566444619a3f5eda57c6480a0, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.358283 6044 ts_tablet_manager.cc:1428] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:23:08.358280 6042 consensus_queue.cc:260] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.358400 5738 heartbeater.cc:503] Master 127.4.78.126:33207 was elected leader, sending a full tablet report...
I20260924 15:23:08.358538 6042 ts_tablet_manager.cc:1428] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:23:08.358436 6043 raft_consensus.cc:348] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.358583 6043 raft_consensus.cc:374] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.358636 6043 raft_consensus.cc:729] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c56c3a54154abbb183d4a39c7406fc, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.358743 5878 heartbeater.cc:503] Master 127.4.78.126:33207 was elected leader, sending a full tablet report...
I20260924 15:23:08.358754 6043 consensus_queue.cc:260] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.359014 6043 ts_tablet_manager.cc:1428] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:23:08.359135 6018 heartbeater.cc:503] Master 127.4.78.126:33207 was elected leader, sending a full tablet report...
W20260924 15:23:08.379045 5879 tablet.cc:2369] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:23:08.454805 5739 tablet.cc:2369] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:08.505399 6050 raft_consensus.cc:482] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:08.505556 6050 raft_consensus.cc:504] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.505966 6050 leader_election.cc:288] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585), c0d7db3566444619a3f5eda57c6480a0 (127.4.78.66:39527)
I20260924 15:23:08.509001 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.506287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:37465 (local address 127.4.78.65:42585)
0924 15:23:08.506334 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.506335 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.506451 (+ 116us) server_negotiation.cc:407] Connection header received
0924 15:23:08.506489 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.506491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.506506 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.506551 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:08.506730 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.507223 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.508241 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.508589 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.508645 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.508656 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.508880 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.508923 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:23:08.509033 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.506215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:40065 (local address 127.4.78.66:39527)
0924 15:23:08.506285 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.506288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.506299 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:23:08.506332 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.506333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.506346 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.506408 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:08.506638 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.507145 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.508135 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.508662 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.508733 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.508745 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.508901 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.508942 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:23:08.509054 5911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.506144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:39527 (local address 127.4.78.67:40065)
0924 15:23:08.506206 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.506220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.506270 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.506431 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.506432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.506434 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:08.506521 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.506523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.507229 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.507230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.508053 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.508055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.508744 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.508745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.508829 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.508833 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.508888 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.508926 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":24}
I20260924 15:23:08.509090 6051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.506164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:42585 (local address 127.4.78.67:37465)
0924 15:23:08.506355 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.506366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.506445 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.506571 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.506572 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.506574 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:08.506650 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.506652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.507305 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.507306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.508156 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.508158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.508665 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.508666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.508750 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.508754 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.508842 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.508906 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":43,"thread_start_us":93,"threads_started":1}
I20260924 15:23:08.509370 5832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441214bf4a4b4bbfb7c75d50e4408421" candidate_uuid: "e7c56c3a54154abbb183d4a39c7406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d7db3566444619a3f5eda57c6480a0" is_pre_election: true
I20260924 15:23:08.509420 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441214bf4a4b4bbfb7c75d50e4408421" candidate_uuid: "e7c56c3a54154abbb183d4a39c7406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" is_pre_election: true
I20260924 15:23:08.509536 5692 raft_consensus.cc:2454] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c56c3a54154abbb183d4a39c7406fc in term 0.
I20260924 15:23:08.509536 5832 raft_consensus.cc:2454] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c56c3a54154abbb183d4a39c7406fc in term 0.
I20260924 15:23:08.509747 5906 leader_election.cc:302] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0d7db3566444619a3f5eda57c6480a0, e7c56c3a54154abbb183d4a39c7406fc; no voters:
I20260924 15:23:08.509907 6050 raft_consensus.cc:2781] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:08.509965 6050 raft_consensus.cc:482] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:08.509982 6050 raft_consensus.cc:3037] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.510962 6050 raft_consensus.cc:504] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.511111 6050 leader_election.cc:288] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [CANDIDATE]: Term 1 election: Requested vote from peers 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585), c0d7db3566444619a3f5eda57c6480a0 (127.4.78.66:39527)
I20260924 15:23:08.511271 5832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441214bf4a4b4bbfb7c75d50e4408421" candidate_uuid: "e7c56c3a54154abbb183d4a39c7406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d7db3566444619a3f5eda57c6480a0"
I20260924 15:23:08.511271 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441214bf4a4b4bbfb7c75d50e4408421" candidate_uuid: "e7c56c3a54154abbb183d4a39c7406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a"
I20260924 15:23:08.511348 5832 raft_consensus.cc:3037] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.511348 5692 raft_consensus.cc:3037] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.512207 5692 raft_consensus.cc:2454] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c56c3a54154abbb183d4a39c7406fc in term 1.
I20260924 15:23:08.512200 5832 raft_consensus.cc:2454] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c56c3a54154abbb183d4a39c7406fc in term 1.
I20260924 15:23:08.512372 5904 leader_election.cc:302] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519f9d6b7ecc4ccaa7a6dce60f065c5a, e7c56c3a54154abbb183d4a39c7406fc; no voters:
I20260924 15:23:08.512506 6050 raft_consensus.cc:2781] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:08.512653 6050 raft_consensus.cc:686] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 1 LEADER]: Becoming Leader. State: Replica: e7c56c3a54154abbb183d4a39c7406fc, State: Running, Role: LEADER
I20260924 15:23:08.512760 6050 consensus_queue.cc:237] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.513630 5533 catalog_manager.cc:5495] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc reported cstate change: term changed from 0 to 1, leader changed from <none> to e7c56c3a54154abbb183d4a39c7406fc (127.4.78.67). New cstate: current_term: 1 leader_uuid: "e7c56c3a54154abbb183d4a39c7406fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } health_report { overall_health: HEALTHY } } }
I20260924 15:23:08.532948 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.530336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34738 (local address 127.4.78.67:40399)
0924 15:23:08.530403 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.530405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.530415 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:08.530507 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.530508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.530527 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.530595 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:08.530747 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.531216 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.532109 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.532305 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.532384 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.532624 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:08.532720 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:08.532789 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.532851 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.532873 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:23:08.534737 5692 raft_consensus.cc:1261] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Refusing update from remote peer e7c56c3a54154abbb183d4a39c7406fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:08.534742 5832 raft_consensus.cc:1261] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Refusing update from remote peer e7c56c3a54154abbb183d4a39c7406fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:08.535106 6052 consensus_queue.cc:1035] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:08.535248 6050 consensus_queue.cc:1035] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:08.537313 6055 mvcc.cc:205] Tried to move back new op lower bound from 7332918839435522048 to 7332918839348916224. Current Snapshot: MvccSnapshot[applied={T|T < 7332918839435522048}]
I20260924 15:23:08.541437 5533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:08.541520 5533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:08.542253 5533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2: Sending DeleteTablet for 3 replicas of tablet 441214bf4a4b4bbfb7c75d50e4408421
I20260924 15:23:08.542580 5671 tablet_service.cc:1514] Processing DeleteTablet for tablet 441214bf4a4b4bbfb7c75d50e4408421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:08 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:46000
I20260924 15:23:08.542610 5952 tablet_service.cc:1514] Processing DeleteTablet for tablet 441214bf4a4b4bbfb7c75d50e4408421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:08 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:34736
I20260924 15:23:08.542667 5812 tablet_service.cc:1514] Processing DeleteTablet for tablet 441214bf4a4b4bbfb7c75d50e4408421 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:08 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:38804
I20260924 15:23:08.542791 6065 tablet_replica.cc:330] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: stopping tablet replica
I20260924 15:23:08.542886 6065 raft_consensus.cc:2229] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:08.542893 6066 tablet_replica.cc:330] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: stopping tablet replica
I20260924 15:23:08.542982 6065 raft_consensus.cc:2258] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:08.542981 6066 raft_consensus.cc:2229] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:08.542949 5533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:08.543076 5533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:08.543092 6066 raft_consensus.cc:2258] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:08.543404 6065 ts_tablet_manager.cc:1905] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:08.544377 5671 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0f479b87134b0da062c68422f7613b (DEFAULT_TABLE table=test-table [id=6429ac9789604a7691ac46aada2e5248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:08.544391 5952 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0f479b87134b0da062c68422f7613b (DEFAULT_TABLE table=test-table [id=6429ac9789604a7691ac46aada2e5248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:08.544391 5812 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0f479b87134b0da062c68422f7613b (DEFAULT_TABLE table=test-table [id=6429ac9789604a7691ac46aada2e5248]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:08.544493 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0f479b87134b0da062c68422f7613b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.544493 5952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0f479b87134b0da062c68422f7613b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.544499 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d0f479b87134b0da062c68422f7613b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.545076 6065 ts_tablet_manager.cc:1918] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:08.545154 6065 log.cc:1196] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/441214bf4a4b4bbfb7c75d50e4408421
I20260924 15:23:08.545415 6065 ts_tablet_manager.cc:1939] T 441214bf4a4b4bbfb7c75d50e4408421 P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Deleting consensus metadata
I20260924 15:23:08.545733 6044 tablet_bootstrap.cc:492] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Bootstrap starting.
I20260924 15:23:08.545802 6043 tablet_bootstrap.cc:492] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: Bootstrap starting.
I20260924 15:23:08.546110 6042 tablet_bootstrap.cc:492] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: Bootstrap starting.
I20260924 15:23:08.546152 6044 tablet_bootstrap.cc:654] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.546267 6043 tablet_bootstrap.cc:654] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.546522 5519 catalog_manager.cc:4841] TS 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585): tablet 441214bf4a4b4bbfb7c75d50e4408421 (table test-table [id=0770c67c13c8443589bba1695838effc]) successfully deleted
I20260924 15:23:08.546725 6066 ts_tablet_manager.cc:1905] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:08.546867 6042 tablet_bootstrap.cc:654] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.546933 6044 tablet_bootstrap.cc:492] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: No bootstrap required, opened a new log
I20260924 15:23:08.546973 6044 ts_tablet_manager.cc:1397] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:08.547101 6044 raft_consensus.cc:348] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.547180 6044 raft_consensus.cc:374] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.547202 6044 raft_consensus.cc:729] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519f9d6b7ecc4ccaa7a6dce60f065c5a, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.547250 6044 consensus_queue.cc:260] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.547340 6044 ts_tablet_manager.cc:1428] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.548373 6066 ts_tablet_manager.cc:1918] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:08.548446 6066 log.cc:1196] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/441214bf4a4b4bbfb7c75d50e4408421
I20260924 15:23:08.548528 6043 tablet_bootstrap.cc:492] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: No bootstrap required, opened a new log
I20260924 15:23:08.548576 6043 ts_tablet_manager.cc:1397] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:23:08.548689 6042 tablet_bootstrap.cc:492] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: No bootstrap required, opened a new log
I20260924 15:23:08.548700 6066 ts_tablet_manager.cc:1939] T 441214bf4a4b4bbfb7c75d50e4408421 P e7c56c3a54154abbb183d4a39c7406fc: Deleting consensus metadata
I20260924 15:23:08.548728 6042 ts_tablet_manager.cc:1397] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 15:23:08.548734 6043 raft_consensus.cc:348] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.548823 6043 raft_consensus.cc:374] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.548842 6043 raft_consensus.cc:729] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c56c3a54154abbb183d4a39c7406fc, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.548887 6043 consensus_queue.cc:260] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.548884 6042 raft_consensus.cc:348] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.548945 6042 raft_consensus.cc:374] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.548967 6042 raft_consensus.cc:729] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d7db3566444619a3f5eda57c6480a0, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.548966 6043 ts_tablet_manager.cc:1428] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.549028 6042 consensus_queue.cc:260] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.549081 5521 catalog_manager.cc:4841] TS e7c56c3a54154abbb183d4a39c7406fc (127.4.78.67:40399): tablet 441214bf4a4b4bbfb7c75d50e4408421 (table test-table [id=0770c67c13c8443589bba1695838effc]) successfully deleted
I20260924 15:23:08.549111 6042 ts_tablet_manager.cc:1428] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.550050 6067 tablet_replica.cc:330] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: stopping tablet replica
I20260924 15:23:08.550172 6067 raft_consensus.cc:2229] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:08.550280 6067 raft_consensus.cc:2258] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:08.550671 6067 ts_tablet_manager.cc:1905] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:08.552140 6067 ts_tablet_manager.cc:1918] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:08.552222 6067 log.cc:1196] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/441214bf4a4b4bbfb7c75d50e4408421
I20260924 15:23:08.552479 6067 ts_tablet_manager.cc:1939] T 441214bf4a4b4bbfb7c75d50e4408421 P c0d7db3566444619a3f5eda57c6480a0: Deleting consensus metadata
I20260924 15:23:08.553133 5521 catalog_manager.cc:4841] TS c0d7db3566444619a3f5eda57c6480a0 (127.4.78.66:39527): tablet 441214bf4a4b4bbfb7c75d50e4408421 (table test-table [id=0770c67c13c8443589bba1695838effc]) successfully deleted
W20260924 15:23:08.559656 6019 tablet.cc:2369] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:08.722458 6060 raft_consensus.cc:482] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:08.722563 6060 raft_consensus.cc:504] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.722793 6060 leader_election.cc:288] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585), c0d7db3566444619a3f5eda57c6480a0 (127.4.78.66:39527)
I20260924 15:23:08.723007 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0f479b87134b0da062c68422f7613b" candidate_uuid: "e7c56c3a54154abbb183d4a39c7406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" is_pre_election: true
I20260924 15:23:08.723030 5832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0f479b87134b0da062c68422f7613b" candidate_uuid: "e7c56c3a54154abbb183d4a39c7406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d7db3566444619a3f5eda57c6480a0" is_pre_election: true
I20260924 15:23:08.723124 5692 raft_consensus.cc:2454] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c56c3a54154abbb183d4a39c7406fc in term 0.
I20260924 15:23:08.723129 5832 raft_consensus.cc:2454] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7c56c3a54154abbb183d4a39c7406fc in term 0.
I20260924 15:23:08.723304 5904 leader_election.cc:302] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519f9d6b7ecc4ccaa7a6dce60f065c5a, e7c56c3a54154abbb183d4a39c7406fc; no voters:
I20260924 15:23:08.723423 6060 raft_consensus.cc:2781] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:08.723467 6060 raft_consensus.cc:482] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:08.723489 6060 raft_consensus.cc:3037] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.724254 6060 raft_consensus.cc:504] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.724403 6060 leader_election.cc:288] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [CANDIDATE]: Term 1 election: Requested vote from peers 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585), c0d7db3566444619a3f5eda57c6480a0 (127.4.78.66:39527)
I20260924 15:23:08.724576 5832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0f479b87134b0da062c68422f7613b" candidate_uuid: "e7c56c3a54154abbb183d4a39c7406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d7db3566444619a3f5eda57c6480a0"
I20260924 15:23:08.724685 5832 raft_consensus.cc:3037] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.724888 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d0f479b87134b0da062c68422f7613b" candidate_uuid: "e7c56c3a54154abbb183d4a39c7406fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a"
I20260924 15:23:08.724953 5692 raft_consensus.cc:3037] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.725368 5832 raft_consensus.cc:2454] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c56c3a54154abbb183d4a39c7406fc in term 1.
I20260924 15:23:08.725579 5906 leader_election.cc:302] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0d7db3566444619a3f5eda57c6480a0, e7c56c3a54154abbb183d4a39c7406fc; no voters:
I20260924 15:23:08.725656 5692 raft_consensus.cc:2454] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7c56c3a54154abbb183d4a39c7406fc in term 1.
I20260924 15:23:08.725705 6060 raft_consensus.cc:2781] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:08.725778 6060 raft_consensus.cc:686] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 1 LEADER]: Becoming Leader. State: Replica: e7c56c3a54154abbb183d4a39c7406fc, State: Running, Role: LEADER
I20260924 15:23:08.725857 6060 consensus_queue.cc:237] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } }
I20260924 15:23:08.726461 5533 catalog_manager.cc:5495] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc reported cstate change: term changed from 0 to 1, leader changed from <none> to e7c56c3a54154abbb183d4a39c7406fc (127.4.78.67). New cstate: current_term: 1 leader_uuid: "e7c56c3a54154abbb183d4a39c7406fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } health_report { overall_health: HEALTHY } } }
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[5489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263387, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:08.753667 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.742470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:33207 (local address 127.0.0.1:50746)
0924 15:23:08.742669 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.742678 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.742734 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.742897 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.742898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.743193 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:08.743252 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.743255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.743879 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.743880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.744388 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.744389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.744709 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.744710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.744763 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.750119 (+ 5356us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:08.750124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:08.751446 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:08.752813 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.752820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.752822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.753001 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.753172 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:08.753172 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:08.753172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:08.753223 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:08.753425 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:08.753431 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:08.753470 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.753520 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.753603 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260924 15:23:08.753726 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.742557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50746 (local address 127.4.78.126:33207)
0924 15:23:08.742631 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.742633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.742736 (+ 103us) server_negotiation.cc:407] Connection header received
0924 15:23:08.742811 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.742812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.742822 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.742885 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:08.743323 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.743811 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.744459 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.744643 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.744696 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.751529 (+ 6833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:08.751530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:08.751530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:08.751536 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:08.752741 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.753067 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.753068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.753068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.753116 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.753278 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.753278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.753278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.753371 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:08.753418 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.753552 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.753647 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":48}
I20260924 15:23:08.755613 5533 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:50746:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:23:08.755889 5533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:08.757562 5671 tablet_service.cc:1467] Processing CreateTablet for tablet 9de6c3ad672c412296f088b4929c06ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7735ad2fd1a4456088fc21085c5cc0df]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:08.757594 5812 tablet_service.cc:1467] Processing CreateTablet for tablet 9de6c3ad672c412296f088b4929c06ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7735ad2fd1a4456088fc21085c5cc0df]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:08.757601 5952 tablet_service.cc:1467] Processing CreateTablet for tablet 9de6c3ad672c412296f088b4929c06ca (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7735ad2fd1a4456088fc21085c5cc0df]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:08.757737 5812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de6c3ad672c412296f088b4929c06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.757736 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de6c3ad672c412296f088b4929c06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.757797 5952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9de6c3ad672c412296f088b4929c06ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:08.759037 6043 tablet_bootstrap.cc:492] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: Bootstrap starting.
I20260924 15:23:08.759207 6044 tablet_bootstrap.cc:492] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Bootstrap starting.
I20260924 15:23:08.759207 6042 tablet_bootstrap.cc:492] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: Bootstrap starting.
I20260924 15:23:08.759392 6043 tablet_bootstrap.cc:654] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.759624 6044 tablet_bootstrap.cc:654] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.759624 6042 tablet_bootstrap.cc:654] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:08.760310 6044 tablet_bootstrap.cc:492] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: No bootstrap required, opened a new log
I20260924 15:23:08.760381 6044 ts_tablet_manager.cc:1397] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:08.760393 6043 tablet_bootstrap.cc:492] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: No bootstrap required, opened a new log
I20260924 15:23:08.760437 6043 ts_tablet_manager.cc:1397] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:08.760488 6042 tablet_bootstrap.cc:492] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: No bootstrap required, opened a new log
I20260924 15:23:08.760533 6042 ts_tablet_manager.cc:1397] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:23:08.760532 6044 raft_consensus.cc:348] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.760551 6043 raft_consensus.cc:348] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.760619 6043 raft_consensus.cc:374] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.760622 6044 raft_consensus.cc:374] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.760646 6044 raft_consensus.cc:729] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 519f9d6b7ecc4ccaa7a6dce60f065c5a, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.760646 6043 raft_consensus.cc:729] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c56c3a54154abbb183d4a39c7406fc, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.760689 6043 consensus_queue.cc:260] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.760690 6044 consensus_queue.cc:260] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.760677 6042 raft_consensus.cc:348] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.760742 6042 raft_consensus.cc:374] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:08.760769 6042 raft_consensus.cc:729] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d7db3566444619a3f5eda57c6480a0, State: Initialized, Role: FOLLOWER
I20260924 15:23:08.760787 6044 ts_tablet_manager.cc:1428] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.760795 6043 ts_tablet_manager.cc:1428] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.760807 6042 consensus_queue.cc:260] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.760893 6042 ts_tablet_manager.cc:1428] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:08.760922 6048 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.760994 6048 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:08.760988 6060 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.761073 6060 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:08.761209 6049 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.761297 6049 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:08.808557 6049 raft_consensus.cc:482] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:08.808651 6049 raft_consensus.cc:504] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.809108 6049 leader_election.cc:288] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585), e7c56c3a54154abbb183d4a39c7406fc (127.4.78.67:40399)
I20260924 15:23:08.812251 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.809318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:36937 (local address 127.4.78.65:42585)
0924 15:23:08.809388 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.809391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.809440 (+ 49us) server_negotiation.cc:407] Connection header received
0924 15:23:08.809525 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.809526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.809537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.809599 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:08.809824 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.810416 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.811468 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.811869 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.811949 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.811978 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.812151 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.812187 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:23:08.812289 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.809240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:42585 (local address 127.4.78.66:36937)
0924 15:23:08.809386 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.809395 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.809447 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.809620 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.809621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.809623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:08.809722 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.809724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.810500 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.810501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.811376 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.811378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.811974 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.811975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.812066 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.812070 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.812133 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.812176 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":37,"thread_start_us":65,"threads_started":1}
I20260924 15:23:08.812314 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.809374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:42517 (local address 127.4.78.67:40399)
0924 15:23:08.809421 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.809422 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.809431 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:08.809526 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.809527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.809537 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.809599 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:08.809861 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.810419 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.811481 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.811908 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.811987 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.811997 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.812197 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.812235 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:23:08.812528 6081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.809240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40399 (local address 127.4.78.66:42517)
0924 15:23:08.809371 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.809384 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.809447 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.809620 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.809621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.809625 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:08.809729 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.809731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.810503 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.810504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.811390 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.811392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.812052 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.812059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.812122 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.812125 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.812192 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.812232 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":61,"thread_start_us":67,"threads_started":1}
I20260924 15:23:08.812566 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de6c3ad672c412296f088b4929c06ca" candidate_uuid: "c0d7db3566444619a3f5eda57c6480a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" is_pre_election: true
I20260924 15:23:08.812664 5692 raft_consensus.cc:2454] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0d7db3566444619a3f5eda57c6480a0 in term 0.
I20260924 15:23:08.812749 5972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de6c3ad672c412296f088b4929c06ca" candidate_uuid: "c0d7db3566444619a3f5eda57c6480a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c56c3a54154abbb183d4a39c7406fc" is_pre_election: true
I20260924 15:23:08.812860 5764 leader_election.cc:302] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 519f9d6b7ecc4ccaa7a6dce60f065c5a, c0d7db3566444619a3f5eda57c6480a0; no voters:
I20260924 15:23:08.812907 5972 raft_consensus.cc:2454] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0d7db3566444619a3f5eda57c6480a0 in term 0.
I20260924 15:23:08.812966 6049 raft_consensus.cc:2781] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:08.813004 6049 raft_consensus.cc:482] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:08.813036 6049 raft_consensus.cc:3037] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.813738 6049 raft_consensus.cc:504] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.813897 6049 leader_election.cc:288] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [CANDIDATE]: Term 1 election: Requested vote from peers 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585), e7c56c3a54154abbb183d4a39c7406fc (127.4.78.67:40399)
I20260924 15:23:08.814056 5972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de6c3ad672c412296f088b4929c06ca" candidate_uuid: "c0d7db3566444619a3f5eda57c6480a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c56c3a54154abbb183d4a39c7406fc"
I20260924 15:23:08.814064 5692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9de6c3ad672c412296f088b4929c06ca" candidate_uuid: "c0d7db3566444619a3f5eda57c6480a0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a"
I20260924 15:23:08.814127 5972 raft_consensus.cc:3037] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.814128 5692 raft_consensus.cc:3037] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:08.814870 5692 raft_consensus.cc:2454] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0d7db3566444619a3f5eda57c6480a0 in term 1.
I20260924 15:23:08.814884 5972 raft_consensus.cc:2454] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0d7db3566444619a3f5eda57c6480a0 in term 1.
I20260924 15:23:08.815042 5766 leader_election.cc:302] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0d7db3566444619a3f5eda57c6480a0, e7c56c3a54154abbb183d4a39c7406fc; no voters:
I20260924 15:23:08.815135 6049 raft_consensus.cc:2781] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:08.815260 6049 raft_consensus.cc:686] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [term 1 LEADER]: Becoming Leader. State: Replica: c0d7db3566444619a3f5eda57c6480a0, State: Running, Role: LEADER
I20260924 15:23:08.815342 6049 consensus_queue.cc:237] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } }
I20260924 15:23:08.815687 6083 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: TxnStatusTablet state changed. Reason: New leader c0d7db3566444619a3f5eda57c6480a0. Latest consensus state: current_term: 1 leader_uuid: "c0d7db3566444619a3f5eda57c6480a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.815812 6083 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:08.816031 6085 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:23:08.816148 6085 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:23:08.816067 5532 catalog_manager.cc:5495] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0d7db3566444619a3f5eda57c6480a0 (127.4.78.66). New cstate: current_term: 1 leader_uuid: "c0d7db3566444619a3f5eda57c6480a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } health_report { overall_health: HEALTHY } } }
I20260924 15:23:08.836097 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.833474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38812 (local address 127.4.78.66:39527)
0924 15:23:08.833530 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.833532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.833540 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:08.833571 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.833571 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.833593 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.833647 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:08.833794 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.834292 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.835361 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.835537 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.835591 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.835782 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:08.835869 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:08.835937 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.836020 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.836037 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:23:08.836153 6077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.833324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:39527 (local address 127.0.0.1:38812)
0924 15:23:08.833387 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.833400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.833456 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.833675 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.833676 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.833678 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:08.833730 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.833732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.834368 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.834369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.835273 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.835276 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.835611 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.835612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.835678 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.835696 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:08.835950 (+ 254us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:08.835958 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.836017 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.836069 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 15:23:08.837195 5692 raft_consensus.cc:1261] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Refusing update from remote peer c0d7db3566444619a3f5eda57c6480a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:08.837299 5972 raft_consensus.cc:1261] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc [term 1 FOLLOWER]: Refusing update from remote peer c0d7db3566444619a3f5eda57c6480a0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:08.837427 6083 consensus_queue.cc:1035] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:08.837536 6049 consensus_queue.cc:1035] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:08.838208 6060 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: TxnStatusTablet state changed. Reason: New leader c0d7db3566444619a3f5eda57c6480a0. Latest consensus state: current_term: 1 leader_uuid: "c0d7db3566444619a3f5eda57c6480a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.838323 6060 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:08.838737 6048 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: TxnStatusTablet state changed. Reason: New leader c0d7db3566444619a3f5eda57c6480a0. Latest consensus state: current_term: 1 leader_uuid: "c0d7db3566444619a3f5eda57c6480a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.838857 6048 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:08.838937 6049 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0d7db3566444619a3f5eda57c6480a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.839021 6049 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:08.839107 6057 mvcc.cc:205] Tried to move back new op lower bound from 7332918840675155968 to 7332918840588189696. Current Snapshot: MvccSnapshot[applied={T|T < 7332918840675155968 or (T in {7332918840675155968})}]
I20260924 15:23:08.839339 6049 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c0d7db3566444619a3f5eda57c6480a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.839408 6049 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P c0d7db3566444619a3f5eda57c6480a0: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:08.839422 6060 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d7db3566444619a3f5eda57c6480a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.839498 6060 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P e7c56c3a54154abbb183d4a39c7406fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:08.840155 6048 tablet_replica.cc:439] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c0d7db3566444619a3f5eda57c6480a0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 } } peers { permanent_uuid: "e7c56c3a54154abbb183d4a39c7406fc" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40399 } } peers { permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 } } }
I20260924 15:23:08.840242 6048 tablet_replica.cc:442] T 9de6c3ad672c412296f088b4929c06ca P 519f9d6b7ecc4ccaa7a6dce60f065c5a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:08.845229 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.842673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:43853 (local address 127.4.78.66:39527)
0924 15:23:08.842729 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.842731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.842888 (+ 157us) server_negotiation.cc:407] Connection header received
0924 15:23:08.842943 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.842943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.842953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.842991 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:08.843109 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.843534 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.844630 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.844797 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.844845 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.845030 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:08.845053 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:08.845094 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.845170 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.845184 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:23:08.845306 6092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.842581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:39527 (local address 127.4.78.67:43853)
0924 15:23:08.842810 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.842822 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.842886 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.843004 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.843004 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.843006 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:08.843055 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.843057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.843605 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.843606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.844551 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.844554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.844875 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.844876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.844938 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.844953 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:08.845114 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:08.845116 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.845169 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.845218 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260924 15:23:08.847208 5832 raft_consensus.cc:1261] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Refusing update from remote peer e7c56c3a54154abbb183d4a39c7406fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:08.847218 5692 raft_consensus.cc:1261] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Refusing update from remote peer e7c56c3a54154abbb183d4a39c7406fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:08.847406 6050 consensus_queue.cc:1035] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "519f9d6b7ecc4ccaa7a6dce60f065c5a" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:08.847492 6060 consensus_queue.cc:1035] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d7db3566444619a3f5eda57c6480a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 39527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:08.854655 6056 mvcc.cc:205] Tried to move back new op lower bound from 7332918840716894208 to 7332918840221450240. Current Snapshot: MvccSnapshot[applied={T|T < 7332918840716894208}]
I20260924 15:23:08.855842 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.852707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:47939 (local address 127.4.78.67:40399)
0924 15:23:08.852741 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.852743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.852751 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:08.852798 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.852799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.852808 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.852848 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:08.852978 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.853403 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.855171 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.855406 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.855457 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.855644 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:08.855668 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:08.855705 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.855757 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.855774 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:23:08.855906 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.852521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40399 (local address 127.4.78.66:47939)
0924 15:23:08.852682 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:08.852693 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:08.852739 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:08.852859 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:08.852860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:08.852861 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:08.852924 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.852925 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.853470 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.853471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.854718 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:08.854721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.855478 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.855479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:08.855550 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.855564 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:08.855706 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:08.855708 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:23:08.855755 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:23:08.855793 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260924 15:23:08.864208 5532 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:08.864310 5532 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:08.864892 5532 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2d29649bf8554083b3bd840ff88b4fe2: Sending DeleteTablet for 3 replicas of tablet 4d0f479b87134b0da062c68422f7613b
I20260924 15:23:08.865157 5671 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0f479b87134b0da062c68422f7613b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:08 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:46000
I20260924 15:23:08.865150 5952 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0f479b87134b0da062c68422f7613b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:08 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:34736
I20260924 15:23:08.865239 6066 tablet_replica.cc:330] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: stopping tablet replica
I20260924 15:23:08.865240 6065 tablet_replica.cc:330] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: stopping tablet replica
I20260924 15:23:08.865288 6065 raft_consensus.cc:2229] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:08.865294 6066 raft_consensus.cc:2229] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:08.865330 6065 raft_consensus.cc:2258] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:08.865345 6066 raft_consensus.cc:2258] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:08.865664 6065 ts_tablet_manager.cc:1905] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:08.865729 6066 ts_tablet_manager.cc:1905] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:08.866376 5812 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d0f479b87134b0da062c68422f7613b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:08 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:38804
I20260924 15:23:08.866460 6067 tablet_replica.cc:330] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: stopping tablet replica
I20260924 15:23:08.866501 6067 raft_consensus.cc:2229] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:08.866531 6067 raft_consensus.cc:2258] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:08.867460 6065 ts_tablet_manager.cc:1918] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:08.867512 6067 ts_tablet_manager.cc:1905] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:08.867538 6065 log.cc:1196] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/4d0f479b87134b0da062c68422f7613b
I20260924 15:23:08.867545 6066 ts_tablet_manager.cc:1918] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:08.867602 6066 log.cc:1196] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/4d0f479b87134b0da062c68422f7613b
I20260924 15:23:08.867799 6065 ts_tablet_manager.cc:1939] T 4d0f479b87134b0da062c68422f7613b P 519f9d6b7ecc4ccaa7a6dce60f065c5a: Deleting consensus metadata
I20260924 15:23:08.867830 6066 ts_tablet_manager.cc:1939] T 4d0f479b87134b0da062c68422f7613b P e7c56c3a54154abbb183d4a39c7406fc: Deleting consensus metadata
I20260924 15:23:08.868101 5521 catalog_manager.cc:4841] TS e7c56c3a54154abbb183d4a39c7406fc (127.4.78.67:40399): tablet 4d0f479b87134b0da062c68422f7613b (table test-table [id=6429ac9789604a7691ac46aada2e5248]) successfully deleted
I20260924 15:23:08.868202 5519 catalog_manager.cc:4841] TS 519f9d6b7ecc4ccaa7a6dce60f065c5a (127.4.78.65:42585): tablet 4d0f479b87134b0da062c68422f7613b (table test-table [id=6429ac9789604a7691ac46aada2e5248]) successfully deleted
I20260924 15:23:08.869036 6067 ts_tablet_manager.cc:1918] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:08.869098 6067 log.cc:1196] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/4d0f479b87134b0da062c68422f7613b
I20260924 15:23:08.869314 6067 ts_tablet_manager.cc:1939] T 4d0f479b87134b0da062c68422f7613b P c0d7db3566444619a3f5eda57c6480a0: Deleting consensus metadata
I20260924 15:23:08.869913 5521 catalog_manager.cc:4841] TS c0d7db3566444619a3f5eda57c6480a0 (127.4.78.66:39527): tablet 4d0f479b87134b0da062c68422f7613b (table test-table [id=6429ac9789604a7691ac46aada2e5248]) successfully deleted
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[5489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:08.876780 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.867373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.4.78.65:42585)
0924 15:23:08.867414 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.867415 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.868379 (+ 964us) server_negotiation.cc:407] Connection header received
0924 15:23:08.868465 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.868466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.868478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.868530 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:08.869113 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.869693 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.870524 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.870733 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.870811 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.873940 (+ 3129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:08.873949 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:08.873949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:08.873956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:08.875594 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.875964 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.875967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.875968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.876039 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.876245 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.876246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.876247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.876385 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:08.876534 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.876602 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.876681 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 15:23:08.877035 5730 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46010
I20260924 15:23:08.884052 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:08.877949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50762 (local address 127.4.78.126:33207)
0924 15:23:08.877991 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:08.877993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:08.878042 (+ 49us) server_negotiation.cc:407] Connection header received
0924 15:23:08.878103 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:08.878104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:08.878114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:08.878176 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:08.878665 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.879454 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.880167 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:08.880480 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:08.880551 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:08.881650 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:08.881651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:08.881652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:08.881659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:08.883121 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.883415 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.883416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.883417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.883475 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:08.883608 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:08.883608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:08.883609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:08.883723 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:08.883809 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:08.883891 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:23:08.883963 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:23:08.884287 5532 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50762
I20260924 15:23:08.884929 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 5606
I20260924 15:23:08.892024 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 5747
I20260924 15:23:08.896584 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 5887
I20260924 15:23:08.901856 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 5502
2026-09-24T15:23:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2434 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[6105](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[6105](info): set up 2 sockets
Sep 24 15:23:08 dist-test-slave-l766 krb5kdc[6105](info): commencing operation
krb5kdc: starting...
W20260924 15:23:09.940524 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:10.056533 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:38397
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:38397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:10.136920 6118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:10.137159 6118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:10.137197 6118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:10.138365 6118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:10.138432 6118 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:23:10.138454 6118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:10.138475 6118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:10.138506 6118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:10.138531 6118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:10.139992 6118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:38397
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:38397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:10.140270 6118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:10.140491 6118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:10.145259 6124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:10.145246 6126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:10.145300 6123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:10.145440 6118 server_base.cc:1034] running on GCE node
I20260924 15:23:10.145712 6118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:10.146030 6118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:10.147176 6118 hybrid_clock.cc:648] HybridClock initialized: now 1790263390147163 us; error 28 us; skew 500 ppm
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:10.149711 6118 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:10.150199 6118 webserver.cc:460] Webserver started at http://127.4.78.126:42129/ using document root <none> and password file <none>
I20260924 15:23:10.150429 6118 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:10.150472 6118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:10.150602 6118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:10.151618 6118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "6ada7bca1b634b83a9402df2d5c0258c"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "e87a8b57f3dc930a9a8ca5548ff7be0c"
server_key_iv: "da936e1ece5a428aa6fb86f3011752b3"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.152006 6118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "6ada7bca1b634b83a9402df2d5c0258c"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "e87a8b57f3dc930a9a8ca5548ff7be0c"
server_key_iv: "da936e1ece5a428aa6fb86f3011752b3"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.153327 6118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:10.154119 6133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.154309 6118 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:10.154390 6118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "6ada7bca1b634b83a9402df2d5c0258c"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "e87a8b57f3dc930a9a8ca5548ff7be0c"
server_key_iv: "da936e1ece5a428aa6fb86f3011752b3"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.154443 6118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:10.162428 6118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:10.164566 6118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:10.164673 6118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:10.169062 6118 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:38397
I20260924 15:23:10.169140 6195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:38397 every 8 connection(s)
I20260924 15:23:10.169467 6118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:10.170190 6196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.170925 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 6118
I20260924 15:23:10.171001 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:10.171170 4409 external_mini_cluster.cc:1428] Setting key c250a17dd9f6b920b0a68f7ea5dd9426
I20260924 15:23:10.173249 6196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: Bootstrap starting.
I20260924 15:23:10.173950 6196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.174278 6196 log.cc:824] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:10.174996 6196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: No bootstrap required, opened a new log
I20260924 15:23:10.176890 6196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada7bca1b634b83a9402df2d5c0258c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38397 } }
I20260924 15:23:10.177044 6196 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.177069 6196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ada7bca1b634b83a9402df2d5c0258c, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.177158 6196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada7bca1b634b83a9402df2d5c0258c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38397 } }
I20260924 15:23:10.177201 6196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:10.177222 6196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:10.177246 6196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.177866 6196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada7bca1b634b83a9402df2d5c0258c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38397 } }
I20260924 15:23:10.177992 6196 leader_election.cc:302] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ada7bca1b634b83a9402df2d5c0258c; no voters:
I20260924 15:23:10.178201 6196 leader_election.cc:288] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:10.178239 6201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:10.178362 6201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [term 1 LEADER]: Becoming Leader. State: Replica: 6ada7bca1b634b83a9402df2d5c0258c, State: Running, Role: LEADER
I20260924 15:23:10.178447 6201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada7bca1b634b83a9402df2d5c0258c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38397 } }
I20260924 15:23:10.178474 6196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:10.178894 6201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ada7bca1b634b83a9402df2d5c0258c. Latest consensus state: current_term: 1 leader_uuid: "6ada7bca1b634b83a9402df2d5c0258c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada7bca1b634b83a9402df2d5c0258c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38397 } } }
I20260924 15:23:10.178984 6201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:10.178910 6202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ada7bca1b634b83a9402df2d5c0258c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ada7bca1b634b83a9402df2d5c0258c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38397 } } }
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:10.179165 6202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:10.179207 6206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:10.179829 6206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:10.183266 6206 catalog_manager.cc:1347] Generated new cluster ID: 5a6b601e02184fcaad048af78c772fe3
I20260924 15:23:10.183321 6206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:10.183753 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.171683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.4.78.126:38397)
0924 15:23:10.171862 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.171871 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.171907 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:23:10.172487 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.172496 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.172562 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.172714 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.173290 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.174673 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.175481 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.176726 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.176848 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.179756 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.179760 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.179760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.179772 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.181920 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.182379 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.182382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.182383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.182496 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.182738 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.182739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.182740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.183053 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.183208 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.183361 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.183509 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260924 15:23:10.208374 6206 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:10.209044 6206 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:10.216670 6206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: Generated new TSK 0
I20260924 15:23:10.216878 6206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:10.243983 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38397
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:10.323832 6222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:10.324048 6222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:10.324072 6222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:10.325222 6222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:10.325263 6222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:10.325284 6222 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:23:10.325330 6222 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:10.325356 6222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:10.326838 6222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:10.327076 6222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:10.327306 6222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:10.332331 6230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:10.332352 6227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:10.332427 6222 server_base.cc:1034] running on GCE node
W20260924 15:23:10.332370 6228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:10.332652 6222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:10.332901 6222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:10.334069 6222 hybrid_clock.cc:648] HybridClock initialized: now 1790263390334049 us; error 37 us; skew 500 ppm
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:10.336457 6222 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:10.336871 6222 webserver.cc:460] Webserver started at http://127.4.78.65:36675/ using document root <none> and password file <none>
I20260924 15:23:10.337103 6222 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:10.337153 6222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:10.337322 6222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:10.338423 6222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "23bfbc8f16ad4e6590718543119cdf53"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "1c4501d9362cb8ba5f41dc060f49aa17"
server_key_iv: "c87774b9e9c66bff3148975bd5a1f966"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.338765 6222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "23bfbc8f16ad4e6590718543119cdf53"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "1c4501d9362cb8ba5f41dc060f49aa17"
server_key_iv: "c87774b9e9c66bff3148975bd5a1f966"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.340018 6222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:10.340701 6237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.340905 6222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:10.340991 6222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "23bfbc8f16ad4e6590718543119cdf53"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "1c4501d9362cb8ba5f41dc060f49aa17"
server_key_iv: "c87774b9e9c66bff3148975bd5a1f966"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.341045 6222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:10.357069 6222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:10.359484 6222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:10.359620 6222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:10.360071 6222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:10.360111 6222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.360134 6222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:10.360147 6222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.367033 6222 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:36709
I20260924 15:23:10.367105 6352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:36709 every 8 connection(s)
I20260924 15:23:10.367408 6222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:10.368577 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 6222
I20260924 15:23:10.368695 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:10.368876 4409 external_mini_cluster.cc:1428] Setting key 366f2bf31c069290756bf62c2563803d
I20260924 15:23:10.370913 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.360607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38397 (local address 127.4.78.65:36011)
0924 15:23:10.360760 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.360794 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.361291 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.361543 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.361547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.361988 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:10.362301 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.362308 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.363435 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.363436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.364809 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.364812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.365351 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.365352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.365453 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.366127 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:10.366137 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:10.367965 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:10.369547 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.369554 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.369556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.369790 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.370052 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.370053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.370053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.370152 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.370441 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:10.370445 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:10.370557 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.370632 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.370753 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":94,"threads_started":1}
I20260924 15:23:10.370930 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.360605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:36011 (local address 127.4.78.126:38397)
0924 15:23:10.360645 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.360649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.360880 (+ 231us) server_negotiation.cc:407] Connection header received
0924 15:23:10.361407 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.361408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.361422 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.361473 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.362426 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.363303 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.364916 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.365256 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.365342 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.368052 (+ 2710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.368054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.368054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.368061 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.369431 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.369909 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.369910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.369911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.369980 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.370224 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.370225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.370226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.370377 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.370445 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.370699 (+ 254us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.370800 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:23:10.371624 6353 heartbeater.cc:347] Connected to a master server at 127.4.78.126:38397
I20260924 15:23:10.371711 6353 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:10.371901 6353 heartbeater.cc:511] Master 127.4.78.126:38397 requested a full tablet report, sending...
I20260924 15:23:10.372452 6149 ts_manager.cc:194] Registered new tserver with Master: 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709)
I20260924 15:23:10.373402 6149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:36011
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
I20260924 15:23:10.378650 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.372705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:56369 (local address 127.4.78.126:38397)
0924 15:23:10.372746 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.372748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.372869 (+ 121us) server_negotiation.cc:407] Connection header received
0924 15:23:10.372941 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.372942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.372951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.373006 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.373353 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.374068 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.374768 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.374969 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.375051 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.376045 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.376047 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.376047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.376054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.377423 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.377779 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.377780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.377781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.377854 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.378050 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.378051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.378051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.378176 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.378263 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.378474 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.378549 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:10.378670 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.372650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38397 (local address 127.4.78.65:56369)
0924 15:23:10.372795 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.372804 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.372884 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.373010 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.373011 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.373205 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:10.373254 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.373255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.374156 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.374157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.374677 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.374679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.375052 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.375053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.375136 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.375654 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:10.375658 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:10.375924 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:10.377528 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.377529 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.377529 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.377704 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.377921 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.377921 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.377922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.377983 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.378254 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:10.378255 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:10.378303 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.378444 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.378551 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":85,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:10.396176 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38397
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:10.476234 6362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:10.476457 6362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:10.476500 6362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:10.477649 6362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:10.477715 6362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:10.477748 6362 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:23:10.477778 6362 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:10.477826 6362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:10.479429 6362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:10.479679 6362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:10.479918 6362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:10.484783 6368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:10.484783 6367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:10.484841 6370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:10.484908 6362 server_base.cc:1034] running on GCE node
I20260924 15:23:10.485188 6362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:10.485464 6362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:10.486625 6362 hybrid_clock.cc:648] HybridClock initialized: now 1790263390486613 us; error 42 us; skew 500 ppm
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:10.489364 6362 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:10.489794 6362 webserver.cc:460] Webserver started at http://127.4.78.66:43587/ using document root <none> and password file <none>
I20260924 15:23:10.490069 6362 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:10.490161 6362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:10.490319 6362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:10.491353 6362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "0e5f676263a349b5a6e99451b1490cb1"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "f923966894543db1894f87136df7b458"
server_key_iv: "ab7ce63cd4fb7960d2f89023cc8cfb4c"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.491719 6362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "0e5f676263a349b5a6e99451b1490cb1"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "f923966894543db1894f87136df7b458"
server_key_iv: "ab7ce63cd4fb7960d2f89023cc8cfb4c"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.493018 6362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:10.493726 6377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.493918 6362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:10.494017 6362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "0e5f676263a349b5a6e99451b1490cb1"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "f923966894543db1894f87136df7b458"
server_key_iv: "ab7ce63cd4fb7960d2f89023cc8cfb4c"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.494104 6362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:10.525945 6362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:10.527810 6362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:10.527984 6362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:10.528530 6362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:10.528594 6362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.528640 6362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:10.528661 6362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:10.536212 6362 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:43477
I20260924 15:23:10.536309 6492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:43477 every 8 connection(s)
I20260924 15:23:10.536646 6362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:10.538448 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.529128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:60859 (local address 127.4.78.126:38397)
0924 15:23:10.529205 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.529209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.529406 (+ 197us) server_negotiation.cc:407] Connection header received
0924 15:23:10.529935 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.529937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.529952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.530036 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.530775 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.531628 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.532712 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.532917 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.532994 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.535703 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.535705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.535705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.535713 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.537302 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.537658 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.537659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.537659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.537709 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.537961 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.537961 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.537962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.538059 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.538105 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.538325 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.538386 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:23:10.538614 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.529081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38397 (local address 127.4.78.66:60859)
0924 15:23:10.529241 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.529290 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.529774 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.530136 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.530139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.530438 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:10.530697 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.530702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.531739 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.531740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.532605 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.532607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.533000 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.533001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.533136 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.533705 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:10.533716 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:10.535593 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:10.537370 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.537374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.537376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.537585 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.537768 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.537768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.537769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.537909 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.538096 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:10.538099 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:10.538203 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.538282 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.538397 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260924 15:23:10.539070 6493 heartbeater.cc:347] Connected to a master server at 127.4.78.126:38397
I20260924 15:23:10.539175 6493 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:10.539359 6493 heartbeater.cc:511] Master 127.4.78.126:38397 requested a full tablet report, sending...
I20260924 15:23:10.539691 6149 ts_manager.cc:194] Registered new tserver with Master: 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477)
I20260924 15:23:10.540012 6149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:60859
I20260924 15:23:10.540901 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 6362
I20260924 15:23:10.541003 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:10.541221 4409 external_mini_cluster.cc:1428] Setting key d309bc42be7e179ba365ad3947dd9e72
I20260924 15:23:10.545992 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.540182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:51795 (local address 127.4.78.126:38397)
0924 15:23:10.540326 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.540328 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.540336 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:10.540355 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.540355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.540365 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.540424 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.540712 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.541354 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.542258 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.542455 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.542524 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.543433 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.543434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.543434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.543441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.544827 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.545216 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.545217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.545217 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.545263 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.545477 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.545478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.545478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.545600 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.545653 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.545862 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.545936 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260924 15:23:10.546028 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.540073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38397 (local address 127.4.78.66:51795)
0924 15:23:10.540210 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.540222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.540271 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.540434 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.540435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.540585 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:10.540638 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.540639 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.541472 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.541474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.542165 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.542167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.542535 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.542536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.542608 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.543022 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:10.543026 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:10.543352 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:10.544902 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.544903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.544904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.545139 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.545338 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.545339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.545340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.545413 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.545666 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:10.545668 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:10.545721 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.545796 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.545918 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:10.566859 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38397
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:37441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:10.646982 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:10.647218 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:10.647261 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:10.648392 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:10.648459 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:10.648495 6502 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:23:10.648530 6502 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:10.648586 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:10.650146 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:37441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:38397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:10.650403 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:10.650633 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:10.655540 6510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:10.655560 6508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:10.655555 6507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:10.655754 6502 server_base.cc:1034] running on GCE node
I20260924 15:23:10.655946 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:10.656188 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:10.657334 6502 hybrid_clock.cc:648] HybridClock initialized: now 1790263390657318 us; error 32 us; skew 500 ppm
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:10.659780 6502 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:10.660202 6502 webserver.cc:460] Webserver started at http://127.4.78.67:42655/ using document root <none> and password file <none>
I20260924 15:23:10.660388 6502 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:10.660427 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:10.660511 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:10.661439 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "df911a6683e1407ab894603f93c507d3"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "313b33d687cb1e81e7921cd0d0ffd137"
server_key_iv: "cd1617661252a28d13f2ec45b2e260d4"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.661736 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "df911a6683e1407ab894603f93c507d3"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "313b33d687cb1e81e7921cd0d0ffd137"
server_key_iv: "cd1617661252a28d13f2ec45b2e260d4"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.663012 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:10.663635 6517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.663800 6502 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:10.663887 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "df911a6683e1407ab894603f93c507d3"
format_stamp: "Formatted at 2026-09-24 15:23:10 on dist-test-slave-l766"
server_key: "313b33d687cb1e81e7921cd0d0ffd137"
server_key_iv: "cd1617661252a28d13f2ec45b2e260d4"
server_key_version: "encryptionkey@0"
I20260924 15:23:10.663966 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:10.670306 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:10.672227 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:10.672379 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:10.672880 6502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:10.672936 6502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.672972 6502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:10.672995 6502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:10.680615 6502 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:39827
I20260924 15:23:10.680708 6632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:39827 every 8 connection(s)
I20260924 15:23:10.681030 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:10.682412 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.673214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:46119 (local address 127.4.78.126:38397)
0924 15:23:10.673263 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.673266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.673525 (+ 259us) server_negotiation.cc:407] Connection header received
0924 15:23:10.674002 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.674003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.674017 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.674104 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.674634 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.675245 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.676236 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.676442 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.676514 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.679050 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.679052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.679052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.679059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.680812 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.681411 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.681412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.681413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.681476 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.681727 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.681728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.681728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.681909 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.681977 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.682243 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.682322 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:23:10.682578 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.673229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38397 (local address 127.4.78.67:46119)
0924 15:23:10.673394 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.673431 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.673909 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.674142 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.674145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.674375 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:10.674536 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.674540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.675355 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.675355 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.676135 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.676137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.676532 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.676533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.676649 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.677141 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:10.677150 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:10.678952 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:10.680920 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.680924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.680926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.681316 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.681547 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.681548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.681548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.681657 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.681976 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:10.681983 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:10.682115 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.682221 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.682368 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260924 15:23:10.683076 6633 heartbeater.cc:347] Connected to a master server at 127.4.78.126:38397
I20260924 15:23:10.683161 6633 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:10.683327 6633 heartbeater.cc:511] Master 127.4.78.126:38397 requested a full tablet report, sending...
I20260924 15:23:10.683665 6150 ts_manager.cc:194] Registered new tserver with Master: df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827)
I20260924 15:23:10.684165 6150 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:46119
I20260924 15:23:10.688865 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.684036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:53605 (local address 127.4.78.126:38397)
0924 15:23:10.684096 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.684097 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.684217 (+ 120us) server_negotiation.cc:407] Connection header received
0924 15:23:10.684264 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.684265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.684275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.684330 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.684609 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.685059 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.685746 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.685993 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.686072 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.686803 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.686804 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.686804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.686811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.687838 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.688164 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.688165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.688165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.688213 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.688422 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.688422 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.688423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.688522 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.688597 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.688720 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.688782 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:23:10.688880 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.683977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38397 (local address 127.4.78.67:53605)
0924 15:23:10.684151 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.684161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.684223 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.684334 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.684335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.684477 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:10.684528 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.684529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.685152 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.685153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.685655 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.685656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.686073 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.686074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.686155 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.686457 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:10.686460 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:10.686718 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:10.687918 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.687919 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.687920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.688096 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.688293 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.688293 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.688294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.688346 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.688583 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:10.688584 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:10.688626 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.688692 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.688784 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":74,"threads_started":1}
I20260924 15:23:10.691155 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 6502
I20260924 15:23:10.691241 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:10.691419 4409 external_mini_cluster.cc:1428] Setting key 1b1119fcade134abcdb836fafad5fb1d
I20260924 15:23:10.692080 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:10.713555 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.706940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.4.78.126:38397)
0924 15:23:10.707001 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.707002 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.707011 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:10.707061 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.707062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.707071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.707122 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.707653 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.708130 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.708853 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.709041 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.709093 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.711315 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.711317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.711317 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.711323 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.712640 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.712947 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.712948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.712948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.712997 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.713158 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.713158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.713159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.713255 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.713327 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.713437 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.713493 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:23:10.715193 6150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:10.716938 6150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:10.723467 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.719220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:39827 (local address 127.0.0.1:40256)
0924 15:23:10.719712 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.719755 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.719831 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.720149 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.720152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.720157 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:10.720222 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.720224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.721121 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.721122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.722316 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.722319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.723146 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.723147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.723223 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.723238 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.723317 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.723367 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":79,"threads_started":1}
I20260924 15:23:10.723572 6651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.719177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36709 (local address 127.0.0.1:32852)
0924 15:23:10.719713 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.719755 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.719831 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.720149 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.720152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.720154 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:10.720220 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.720222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.721524 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.721525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.722548 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.722550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.723334 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.723335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.723394 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.723398 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.723454 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.723491 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":70,"threads_started":1}
I20260924 15:23:10.723776 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.719307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.4.78.65:36709)
0924 15:23:10.719499 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.719504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.719833 (+ 329us) server_negotiation.cc:407] Connection header received
0924 15:23:10.719963 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.719965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.719982 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.720058 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:10.720301 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.721437 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.722628 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.723261 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.723325 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.723377 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.723479 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.723687 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260924 15:23:10.723886 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.719551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.4.78.67:39827)
0924 15:23:10.719774 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.719777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.719857 (+ 80us) server_negotiation.cc:407] Connection header received
0924 15:23:10.719963 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.719965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.719983 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.720070 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:10.720322 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.721019 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.722404 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.723063 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.723132 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.723186 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.723737 (+ 551us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.723776 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":82,"threads_started":1}
I20260924 15:23:10.723934 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.719397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43477 (local address 127.0.0.1:52466)
0924 15:23:10.720025 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.720036 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.720079 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.720279 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.720280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.720282 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:10.720323 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.720324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.721863 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.721864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.722759 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.722762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.723721 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.723722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.723777 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.723780 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.723828 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.723864 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":85,"threads_started":1}
I20260924 15:23:10.724190 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.719533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.4.78.66:43477)
0924 15:23:10.719709 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.719712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.720132 (+ 420us) server_negotiation.cc:407] Connection header received
0924 15:23:10.720173 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.720175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.720189 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.720268 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:10.720412 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.721450 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.722845 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.723568 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.723632 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.723687 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.724021 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.724093 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
I20260924 15:23:10.724524 6287 tablet_service.cc:1467] Processing CreateTablet for tablet 5a49d0d7ea8942eda839f610380eba8b (DEFAULT_TABLE table=test-table [id=ebe248de36f9437eafdcaeb908f18db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:10.724555 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 5a49d0d7ea8942eda839f610380eba8b (DEFAULT_TABLE table=test-table [id=ebe248de36f9437eafdcaeb908f18db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:10.724700 6427 tablet_service.cc:1467] Processing CreateTablet for tablet 5a49d0d7ea8942eda839f610380eba8b (DEFAULT_TABLE table=test-table [id=ebe248de36f9437eafdcaeb908f18db3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:10.724884 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a49d0d7ea8942eda839f610380eba8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.724906 6287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a49d0d7ea8942eda839f610380eba8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.724979 6427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a49d0d7ea8942eda839f610380eba8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.727665 6656 tablet_bootstrap.cc:492] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: Bootstrap starting.
I20260924 15:23:10.728420 6658 tablet_bootstrap.cc:492] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Bootstrap starting.
I20260924 15:23:10.728477 6656 tablet_bootstrap.cc:654] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.728770 6656 log.cc:824] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:10.728993 6658 tablet_bootstrap.cc:654] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.729159 6657 tablet_bootstrap.cc:492] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: Bootstrap starting.
I20260924 15:23:10.729360 6658 log.cc:824] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:10.729533 6656 tablet_bootstrap.cc:492] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: No bootstrap required, opened a new log
I20260924 15:23:10.729585 6656 ts_tablet_manager.cc:1397] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:10.730094 6658 tablet_bootstrap.cc:492] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: No bootstrap required, opened a new log
I20260924 15:23:10.730176 6658 ts_tablet_manager.cc:1397] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:10.731279 6656 raft_consensus.cc:348] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.731405 6656 raft_consensus.cc:374] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.731426 6656 raft_consensus.cc:729] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bfbc8f16ad4e6590718543119cdf53, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.731541 6656 consensus_queue.cc:260] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.731564 6658 raft_consensus.cc:348] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.731747 6658 raft_consensus.cc:374] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.731796 6658 raft_consensus.cc:729] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5f676263a349b5a6e99451b1490cb1, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.731812 6656 ts_tablet_manager.cc:1428] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:10.731838 6353 heartbeater.cc:503] Master 127.4.78.126:38397 was elected leader, sending a full tablet report...
I20260924 15:23:10.731937 6658 consensus_queue.cc:260] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.732205 6658 ts_tablet_manager.cc:1428] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:10.732255 6493 heartbeater.cc:503] Master 127.4.78.126:38397 was elected leader, sending a full tablet report...
I20260924 15:23:10.732784 6657 tablet_bootstrap.cc:654] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.733115 6657 log.cc:824] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:10.733749 6657 tablet_bootstrap.cc:492] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: No bootstrap required, opened a new log
I20260924 15:23:10.733799 6657 ts_tablet_manager.cc:1397] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 15:23:10.735307 6657 raft_consensus.cc:348] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.735417 6657 raft_consensus.cc:374] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.735437 6657 raft_consensus.cc:729] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df911a6683e1407ab894603f93c507d3, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.735507 6657 consensus_queue.cc:260] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.735683 6657 ts_tablet_manager.cc:1428] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:23:10.735767 6633 heartbeater.cc:503] Master 127.4.78.126:38397 was elected leader, sending a full tablet report...
I20260924 15:23:10.737897 6664 raft_consensus.cc:482] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:10.737974 6664 raft_consensus.cc:504] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.738312 6664 leader_election.cc:288] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709), 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477)
I20260924 15:23:10.742194 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.738656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:36895 (local address 127.4.78.66:43477)
0924 15:23:10.738692 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.738694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.738701 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:10.738723 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.738724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.738733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.738788 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:10.738935 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.739670 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.741279 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.741851 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.741939 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.741948 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.742105 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.742148 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:10.742225 6665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.738456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43477 (local address 127.4.78.67:36895)
0924 15:23:10.738606 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.738615 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.738675 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.738788 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.738789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.738790 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:10.738854 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.738855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.739804 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.739805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.741071 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.741074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.741945 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.741947 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.742020 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.742025 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.742087 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.742135 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":88,"thread_start_us":67,"threads_started":1}
I20260924 15:23:10.742360 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.738497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:42243 (local address 127.4.78.65:36709)
0924 15:23:10.738606 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.738608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.738617 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:10.738690 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.738691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.738703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.738755 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:10.738916 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.739684 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.741293 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.741873 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.741935 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.741946 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.742256 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.742303 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 15:23:10.742226 6526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.738456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36709 (local address 127.4.78.67:42243)
0924 15:23:10.738542 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.738554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.738611 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.738758 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.738758 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.738760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:10.738826 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.738828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.739804 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.739805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.741118 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.741119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.741953 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.741954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.742089 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.742092 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.742138 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.742169 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":32}
I20260924 15:23:10.742833 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a49d0d7ea8942eda839f610380eba8b" candidate_uuid: "df911a6683e1407ab894603f93c507d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1" is_pre_election: true
I20260924 15:23:10.742908 6307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a49d0d7ea8942eda839f610380eba8b" candidate_uuid: "df911a6683e1407ab894603f93c507d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23bfbc8f16ad4e6590718543119cdf53" is_pre_election: true
I20260924 15:23:10.742992 6447 raft_consensus.cc:2454] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df911a6683e1407ab894603f93c507d3 in term 0.
I20260924 15:23:10.743062 6307 raft_consensus.cc:2454] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df911a6683e1407ab894603f93c507d3 in term 0.
I20260924 15:23:10.743242 6521 leader_election.cc:302] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f676263a349b5a6e99451b1490cb1, df911a6683e1407ab894603f93c507d3; no voters:
I20260924 15:23:10.743453 6664 raft_consensus.cc:2781] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:10.743502 6664 raft_consensus.cc:482] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:10.743518 6664 raft_consensus.cc:3037] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.744324 6664 raft_consensus.cc:504] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.744520 6664 leader_election.cc:288] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [CANDIDATE]: Term 1 election: Requested vote from peers 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709), 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477)
I20260924 15:23:10.744581 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a49d0d7ea8942eda839f610380eba8b" candidate_uuid: "df911a6683e1407ab894603f93c507d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1"
I20260924 15:23:10.744629 6307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a49d0d7ea8942eda839f610380eba8b" candidate_uuid: "df911a6683e1407ab894603f93c507d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23bfbc8f16ad4e6590718543119cdf53"
I20260924 15:23:10.744658 6447 raft_consensus.cc:3037] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.744711 6307 raft_consensus.cc:3037] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.745489 6447 raft_consensus.cc:2454] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df911a6683e1407ab894603f93c507d3 in term 1.
I20260924 15:23:10.745584 6307 raft_consensus.cc:2454] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df911a6683e1407ab894603f93c507d3 in term 1.
I20260924 15:23:10.745754 6521 leader_election.cc:302] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f676263a349b5a6e99451b1490cb1, df911a6683e1407ab894603f93c507d3; no voters:
I20260924 15:23:10.745896 6664 raft_consensus.cc:2781] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:10.746013 6664 raft_consensus.cc:686] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 1 LEADER]: Becoming Leader. State: Replica: df911a6683e1407ab894603f93c507d3, State: Running, Role: LEADER
I20260924 15:23:10.746194 6664 consensus_queue.cc:237] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } }
I20260924 15:23:10.746942 6150 catalog_manager.cc:5495] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df911a6683e1407ab894603f93c507d3 (127.4.78.67). New cstate: current_term: 1 leader_uuid: "df911a6683e1407ab894603f93c507d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } health_report { overall_health: UNKNOWN } } }
Sep 24 15:23:10 dist-test-slave-l766 krb5kdc[6105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263390, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:10.773370 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.766549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38397 (local address 127.0.0.1:46752)
0924 15:23:10.766739 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.766749 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.766813 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.766925 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.766926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.767241 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:10.767280 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.767282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.767878 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.767879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.768436 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.768437 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.768753 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.768753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.768793 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.769283 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:10.769288 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:10.770616 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:10.772218 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.772227 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.772229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.772438 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.772640 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:10.772640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:10.772641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:10.772698 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:10.772943 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:10.772950 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:10.772994 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.773099 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.773231 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260924 15:23:10.773377 6199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.766598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.4.78.126:38397)
0924 15:23:10.766658 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.766665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.766814 (+ 149us) server_negotiation.cc:407] Connection header received
0924 15:23:10.766851 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.766851 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.766860 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.766908 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:10.767344 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.767817 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.768507 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.768690 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.768738 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.770714 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:10.770716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:10.770716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:10.770723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:10.772081 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.772514 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.772515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.772516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.772574 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:10.772766 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:10.772767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:10.772767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:10.772878 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:10.772951 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.773122 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.773299 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":35}
I20260924 15:23:10.775437 6150 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:46752:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:23:10.775643 6150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:10.777155 6427 tablet_service.cc:1467] Processing CreateTablet for tablet ffb2584da937462691511eeaa9a3865d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a527f1ba8ca405ebae152cdcb54e5c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:10.777165 6287 tablet_service.cc:1467] Processing CreateTablet for tablet ffb2584da937462691511eeaa9a3865d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a527f1ba8ca405ebae152cdcb54e5c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:10.777155 6567 tablet_service.cc:1467] Processing CreateTablet for tablet ffb2584da937462691511eeaa9a3865d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9a527f1ba8ca405ebae152cdcb54e5c9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:10.777315 6427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb2584da937462691511eeaa9a3865d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.777323 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb2584da937462691511eeaa9a3865d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.777339 6287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffb2584da937462691511eeaa9a3865d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.778596 6656 tablet_bootstrap.cc:492] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: Bootstrap starting.
I20260924 15:23:10.778915 6658 tablet_bootstrap.cc:492] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: Bootstrap starting.
I20260924 15:23:10.779211 6657 tablet_bootstrap.cc:492] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: Bootstrap starting.
I20260924 15:23:10.779433 6656 tablet_bootstrap.cc:654] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.779651 6657 tablet_bootstrap.cc:654] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.779649 6658 tablet_bootstrap.cc:654] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.780139 6656 tablet_bootstrap.cc:492] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: No bootstrap required, opened a new log
I20260924 15:23:10.780210 6656 ts_tablet_manager.cc:1397] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:10.780323 6657 tablet_bootstrap.cc:492] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: No bootstrap required, opened a new log
I20260924 15:23:10.780377 6657 ts_tablet_manager.cc:1397] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:10.780366 6656 raft_consensus.cc:348] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.780440 6656 raft_consensus.cc:374] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.780467 6656 raft_consensus.cc:729] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bfbc8f16ad4e6590718543119cdf53, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.780476 6658 tablet_bootstrap.cc:492] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: No bootstrap required, opened a new log
I20260924 15:23:10.780529 6658 ts_tablet_manager.cc:1397] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:10.780534 6656 consensus_queue.cc:260] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.780545 6657 raft_consensus.cc:348] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.780612 6657 raft_consensus.cc:374] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.780637 6657 raft_consensus.cc:729] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df911a6683e1407ab894603f93c507d3, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.780653 6656 ts_tablet_manager.cc:1428] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:10.780687 6657 consensus_queue.cc:260] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.780732 6658 raft_consensus.cc:348] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.780783 6657 ts_tablet_manager.cc:1428] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:10.780817 6658 raft_consensus.cc:374] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.780844 6658 raft_consensus.cc:729] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5f676263a349b5a6e99451b1490cb1, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.780807 6661 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.780901 6661 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:10.780900 6664 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.780913 6658 consensus_queue.cc:260] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.780961 6664 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:10.781052 6662 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.781148 6662 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:10.781358 6658 ts_tablet_manager.cc:1428] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:10.786302 6661 raft_consensus.cc:482] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:10.786371 6661 raft_consensus.cc:504] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.786654 6661 leader_election.cc:288] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477), df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827)
W20260924 15:23:10.787257 6494 tablet.cc:2369] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:10.789492 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.787089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:57491 (local address 127.4.78.66:43477)
0924 15:23:10.787132 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.787135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.787144 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:10.787167 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.787168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.787177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.787264 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:10.787417 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.787861 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.788755 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.789122 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.789224 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.789234 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.789405 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.789439 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:23:10.789505 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.786815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43477 (local address 127.4.78.65:57491)
0924 15:23:10.786912 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.786925 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.786972 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.787283 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.787284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.787286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:10.787355 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.787356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.787957 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.787958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.788658 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.788660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.789237 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.789238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.789324 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.789327 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.789388 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.789427 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 15:23:10.789772 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb2584da937462691511eeaa9a3865d" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1" is_pre_election: true
I20260924 15:23:10.789849 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.787147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:44447 (local address 127.4.78.67:39827)
0924 15:23:10.787193 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.787195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.787203 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:10.787541 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.787542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.787553 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.787610 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:10.787757 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.788207 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.789100 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.789457 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.789549 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.789557 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.789741 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.789775 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:23:10.789870 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.786956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:39827 (local address 127.4.78.65:44447)
0924 15:23:10.787088 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.787097 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.787134 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.787616 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.787617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.787619 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:10.787685 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.787686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.788305 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.788306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.788982 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.788984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.789555 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.789556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.789644 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.789649 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.789731 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.789775 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260924 15:23:10.789891 6447 raft_consensus.cc:2454] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 0.
I20260924 15:23:10.790105 6241 leader_election.cc:302] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f676263a349b5a6e99451b1490cb1, 23bfbc8f16ad4e6590718543119cdf53; no voters:
I20260924 15:23:10.790249 6661 raft_consensus.cc:2781] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:10.790304 6661 raft_consensus.cc:482] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:10.790330 6661 raft_consensus.cc:3037] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.790405 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb2584da937462691511eeaa9a3865d" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df911a6683e1407ab894603f93c507d3" is_pre_election: true
I20260924 15:23:10.790521 6587 raft_consensus.cc:2454] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 0.
I20260924 15:23:10.791074 6661 raft_consensus.cc:504] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.791168 6661 leader_election.cc:288] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 election: Requested vote from peers 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477), df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827)
I20260924 15:23:10.791342 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb2584da937462691511eeaa9a3865d" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df911a6683e1407ab894603f93c507d3"
I20260924 15:23:10.791368 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffb2584da937462691511eeaa9a3865d" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1"
I20260924 15:23:10.791420 6447 raft_consensus.cc:3037] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.791420 6587 raft_consensus.cc:3037] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.792112 6447 raft_consensus.cc:2454] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 1.
I20260924 15:23:10.792114 6587 raft_consensus.cc:2454] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 1.
I20260924 15:23:10.792285 6241 leader_election.cc:302] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f676263a349b5a6e99451b1490cb1, 23bfbc8f16ad4e6590718543119cdf53; no voters:
I20260924 15:23:10.792375 6661 raft_consensus.cc:2781] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:10.792518 6661 raft_consensus.cc:686] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [term 1 LEADER]: Becoming Leader. State: Replica: 23bfbc8f16ad4e6590718543119cdf53, State: Running, Role: LEADER
I20260924 15:23:10.792611 6661 consensus_queue.cc:237] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.793040 6678 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: TxnStatusTablet state changed. Reason: New leader 23bfbc8f16ad4e6590718543119cdf53. Latest consensus state: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.793152 6678 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:10.793363 6680 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:23:10.793334 6150 catalog_manager.cc:5495] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } health_report { overall_health: UNKNOWN } } }
I20260924 15:23:10.793468 6680 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:23:10.799688 6673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.796858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36709 (local address 127.0.0.1:32868)
0924 15:23:10.796912 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.796923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.796973 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.797196 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.797196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.797197 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:10.797239 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.797241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.798157 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.798158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.798720 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.798722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.799241 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.799241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.799292 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.799305 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:10.799572 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:10.799578 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.799606 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.799638 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 15:23:10.799703 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.796906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32868 (local address 127.4.78.65:36709)
0924 15:23:10.796950 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.796952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.796962 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:10.797063 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.797064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.797082 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.797149 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:10.797320 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.798073 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.798808 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.799162 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.799235 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.799392 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:10.799508 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:10.799592 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.799610 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.799630 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:23:10.801383 6587 raft_consensus.cc:1261] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Refusing update from remote peer 23bfbc8f16ad4e6590718543119cdf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:10.801381 6447 raft_consensus.cc:1261] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Refusing update from remote peer 23bfbc8f16ad4e6590718543119cdf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:10.801656 6678 consensus_queue.cc:1035] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:10.801826 6661 consensus_queue.cc:1035] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:10.802892 6664 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: TxnStatusTablet state changed. Reason: New leader 23bfbc8f16ad4e6590718543119cdf53. Latest consensus state: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.802975 6664 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:10.802996 6662 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: TxnStatusTablet state changed. Reason: New leader 23bfbc8f16ad4e6590718543119cdf53. Latest consensus state: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.803087 6662 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:10.804026 6661 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.804112 6661 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:10.804215 6662 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.804332 6662 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P 0e5f676263a349b5a6e99451b1490cb1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:10.804489 6661 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.804564 6661 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P 23bfbc8f16ad4e6590718543119cdf53: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:10.804860 6685 mvcc.cc:205] Tried to move back new op lower bound from 7332918848719908864 to 7332918848687058944. Current Snapshot: MvccSnapshot[applied={T|T < 7332918848719908864}]
I20260924 15:23:10.804889 6683 mvcc.cc:205] Tried to move back new op lower bound from 7332918848719908864 to 7332918848687058944. Current Snapshot: MvccSnapshot[applied={T|T < 7332918848719908864}]
I20260924 15:23:10.805022 6664 tablet_replica.cc:439] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } } }
I20260924 15:23:10.805100 6664 tablet_replica.cc:442] T ffb2584da937462691511eeaa9a3865d P df911a6683e1407ab894603f93c507d3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:10.805766 6684 mvcc.cc:205] Tried to move back new op lower bound from 7332918848719908864 to 7332918848687058944. Current Snapshot: MvccSnapshot[applied={T|T < 7332918848719908864}]
I20260924 15:23:10.809262 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.806666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.4.78.67:39827)
0924 15:23:10.806713 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.806715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.806727 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:23:10.806792 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.806792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.806809 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.806877 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:10.807040 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.807478 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.808565 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.808733 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.808784 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.808946 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:10.809055 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:10.809121 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.809196 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.809212 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:23:10.814060 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.811660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:57587 (local address 127.4.78.65:36709)
0924 15:23:10.811706 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.811708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.811779 (+ 71us) server_negotiation.cc:407] Connection header received
0924 15:23:10.811871 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.811872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.811884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.811936 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:10.812112 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.812548 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.813296 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.813494 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.813565 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.813749 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:10.813858 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:10.813906 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.813979 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.813998 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 15:23:10.814083 6639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.811566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36709 (local address 127.4.78.67:57587)
0924 15:23:10.811731 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.811742 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.811793 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.811978 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.811978 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.811980 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:10.812033 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.812034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.812627 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.812629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.813188 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.813190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.813577 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.813578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.813647 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.813660 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:10.813908 (+ 248us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:10.813911 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.813974 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.814016 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 15:23:10.816493 6447 raft_consensus.cc:1261] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Refusing update from remote peer df911a6683e1407ab894603f93c507d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:10.816552 6307 raft_consensus.cc:1261] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Refusing update from remote peer df911a6683e1407ab894603f93c507d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:10.816800 6666 consensus_queue.cc:1035] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:10.816924 6664 consensus_queue.cc:1035] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:10.825544 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.822615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:39827 (local address 127.4.78.65:46883)
0924 15:23:10.822667 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:10.822678 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:10.822717 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:10.823314 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:10.823315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:10.823316 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:10.823377 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.823378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.824006 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.824007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.824623 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:10.824625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.825085 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.825086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:10.825160 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.825178 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:10.825381 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:10.825383 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:23:10.825425 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 15:23:10.825472 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 15:23:10.825730 6655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:10.823119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:46883 (local address 127.4.78.67:39827)
0924 15:23:10.823177 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:10.823179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:10.823189 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:10.823211 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:10.823212 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:10.823223 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:10.823273 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:10.823452 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.823918 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.824723 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:10.825025 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:10.825085 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:10.825263 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:10.825298 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:10.825353 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:10.825642 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 15:23:10.825660 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:23:10.832043 6150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:10.832158 6150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:10.832851 6150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: Sending DeleteTablet for 3 replicas of tablet 5a49d0d7ea8942eda839f610380eba8b
I20260924 15:23:10.833122 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a49d0d7ea8942eda839f610380eba8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:10 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:40256
I20260924 15:23:10.833168 6427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a49d0d7ea8942eda839f610380eba8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:10 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:52466
I20260924 15:23:10.833357 6702 tablet_replica.cc:330] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: stopping tablet replica
I20260924 15:23:10.833451 6702 raft_consensus.cc:2229] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:10.833514 6702 raft_consensus.cc:2258] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:10.833623 6287 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a49d0d7ea8942eda839f610380eba8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:10 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:32852
I20260924 15:23:10.833984 6702 ts_tablet_manager.cc:1905] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:10.834092 6150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:10.834228 6150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:10.835320 6703 tablet_replica.cc:330] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: stopping tablet replica
I20260924 15:23:10.835475 6703 raft_consensus.cc:2229] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:10.835603 6703 raft_consensus.cc:2258] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:10.835669 6287 tablet_service.cc:1467] Processing CreateTablet for tablet 57f9afab13b949cfbb7cbd8a796a2411 (DEFAULT_TABLE table=test-table [id=4e2cde53a50a4ffdb0bfe473f55d7555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:10.835811 6287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f9afab13b949cfbb7cbd8a796a2411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.835865 6702 ts_tablet_manager.cc:1918] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:10.835933 6702 log.cc:1196] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/5a49d0d7ea8942eda839f610380eba8b
I20260924 15:23:10.836207 6702 ts_tablet_manager.cc:1939] T 5a49d0d7ea8942eda839f610380eba8b P 0e5f676263a349b5a6e99451b1490cb1: Deleting consensus metadata
I20260924 15:23:10.836211 6703 ts_tablet_manager.cc:1905] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:10.837182 6656 tablet_bootstrap.cc:492] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: Bootstrap starting.
I20260924 15:23:10.837196 6427 tablet_service.cc:1467] Processing CreateTablet for tablet 57f9afab13b949cfbb7cbd8a796a2411 (DEFAULT_TABLE table=test-table [id=4e2cde53a50a4ffdb0bfe473f55d7555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:10.837294 6427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f9afab13b949cfbb7cbd8a796a2411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.837594 6656 tablet_bootstrap.cc:654] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.838186 6656 tablet_bootstrap.cc:492] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: No bootstrap required, opened a new log
I20260924 15:23:10.838241 6656 ts_tablet_manager.cc:1397] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:10.838395 6656 raft_consensus.cc:348] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.838486 6656 raft_consensus.cc:374] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.838515 6656 raft_consensus.cc:729] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bfbc8f16ad4e6590718543119cdf53, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.838541 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 57f9afab13b949cfbb7cbd8a796a2411 (DEFAULT_TABLE table=test-table [id=4e2cde53a50a4ffdb0bfe473f55d7555]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:10.838555 6656 consensus_queue.cc:260] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.838644 6656 ts_tablet_manager.cc:1428] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:10.838654 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f9afab13b949cfbb7cbd8a796a2411. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:10.839529 6703 ts_tablet_manager.cc:1918] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:10.839630 6703 log.cc:1196] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/5a49d0d7ea8942eda839f610380eba8b
I20260924 15:23:10.839866 6703 ts_tablet_manager.cc:1939] T 5a49d0d7ea8942eda839f610380eba8b P 23bfbc8f16ad4e6590718543119cdf53: Deleting consensus metadata
I20260924 15:23:10.839944 6657 tablet_bootstrap.cc:492] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: Bootstrap starting.
I20260924 15:23:10.839959 6658 tablet_bootstrap.cc:492] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: Bootstrap starting.
I20260924 15:23:10.840297 6701 tablet_replica.cc:330] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: stopping tablet replica
I20260924 15:23:10.840342 6658 tablet_bootstrap.cc:654] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.840396 6657 tablet_bootstrap.cc:654] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:10.840396 6701 raft_consensus.cc:2229] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:10.840554 6701 raft_consensus.cc:2258] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:10.840803 6137 catalog_manager.cc:4841] TS 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477): tablet 5a49d0d7ea8942eda839f610380eba8b (table test-table [id=ebe248de36f9437eafdcaeb908f18db3]) successfully deleted
I20260924 15:23:10.841101 6701 ts_tablet_manager.cc:1905] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:10.841140 6136 catalog_manager.cc:4841] TS 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709): tablet 5a49d0d7ea8942eda839f610380eba8b (table test-table [id=ebe248de36f9437eafdcaeb908f18db3]) successfully deleted
I20260924 15:23:10.842890 6701 ts_tablet_manager.cc:1918] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:10.842989 6701 log.cc:1196] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/5a49d0d7ea8942eda839f610380eba8b
I20260924 15:23:10.843215 6701 ts_tablet_manager.cc:1939] T 5a49d0d7ea8942eda839f610380eba8b P df911a6683e1407ab894603f93c507d3: Deleting consensus metadata
I20260924 15:23:10.843672 6657 tablet_bootstrap.cc:492] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: No bootstrap required, opened a new log
I20260924 15:23:10.843722 6657 ts_tablet_manager.cc:1397] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 15:23:10.843796 6137 catalog_manager.cc:4841] TS df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827): tablet 5a49d0d7ea8942eda839f610380eba8b (table test-table [id=ebe248de36f9437eafdcaeb908f18db3]) successfully deleted
I20260924 15:23:10.843863 6657 raft_consensus.cc:348] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.843945 6657 raft_consensus.cc:374] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.843963 6657 raft_consensus.cc:729] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df911a6683e1407ab894603f93c507d3, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.844058 6657 consensus_queue.cc:260] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.844149 6657 ts_tablet_manager.cc:1428] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.844266 6658 tablet_bootstrap.cc:492] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: No bootstrap required, opened a new log
I20260924 15:23:10.844307 6658 ts_tablet_manager.cc:1397] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 15:23:10.844439 6658 raft_consensus.cc:348] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.844534 6658 raft_consensus.cc:374] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:10.844564 6658 raft_consensus.cc:729] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5f676263a349b5a6e99451b1490cb1, State: Initialized, Role: FOLLOWER
I20260924 15:23:10.844609 6658 consensus_queue.cc:260] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.844700 6658 ts_tablet_manager.cc:1428] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:10.857802 6698 raft_consensus.cc:482] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:10.857966 6698 raft_consensus.cc:504] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.858135 6698 leader_election.cc:288] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709), 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477)
I20260924 15:23:10.858309 6307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f9afab13b949cfbb7cbd8a796a2411" candidate_uuid: "df911a6683e1407ab894603f93c507d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23bfbc8f16ad4e6590718543119cdf53" is_pre_election: true
I20260924 15:23:10.858390 6307 raft_consensus.cc:2454] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df911a6683e1407ab894603f93c507d3 in term 0.
I20260924 15:23:10.858412 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f9afab13b949cfbb7cbd8a796a2411" candidate_uuid: "df911a6683e1407ab894603f93c507d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1" is_pre_election: true
I20260924 15:23:10.858482 6447 raft_consensus.cc:2454] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df911a6683e1407ab894603f93c507d3 in term 0.
I20260924 15:23:10.858536 6520 leader_election.cc:302] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23bfbc8f16ad4e6590718543119cdf53, df911a6683e1407ab894603f93c507d3; no voters:
I20260924 15:23:10.858618 6698 raft_consensus.cc:2781] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:10.858646 6698 raft_consensus.cc:482] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:10.858667 6698 raft_consensus.cc:3037] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.859241 6698 raft_consensus.cc:504] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.859380 6698 leader_election.cc:288] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [CANDIDATE]: Term 1 election: Requested vote from peers 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709), 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477)
I20260924 15:23:10.859543 6447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f9afab13b949cfbb7cbd8a796a2411" candidate_uuid: "df911a6683e1407ab894603f93c507d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1"
I20260924 15:23:10.859609 6447 raft_consensus.cc:3037] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.859594 6307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f9afab13b949cfbb7cbd8a796a2411" candidate_uuid: "df911a6683e1407ab894603f93c507d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23bfbc8f16ad4e6590718543119cdf53"
I20260924 15:23:10.859650 6307 raft_consensus.cc:3037] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:10.860306 6447 raft_consensus.cc:2454] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df911a6683e1407ab894603f93c507d3 in term 1.
I20260924 15:23:10.860378 6307 raft_consensus.cc:2454] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df911a6683e1407ab894603f93c507d3 in term 1.
I20260924 15:23:10.860478 6521 leader_election.cc:302] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f676263a349b5a6e99451b1490cb1, df911a6683e1407ab894603f93c507d3; no voters:
I20260924 15:23:10.860654 6698 raft_consensus.cc:2781] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:10.860713 6698 raft_consensus.cc:686] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 1 LEADER]: Becoming Leader. State: Replica: df911a6683e1407ab894603f93c507d3, State: Running, Role: LEADER
I20260924 15:23:10.860769 6698 consensus_queue.cc:237] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:10.861274 6150 catalog_manager.cc:5495] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to df911a6683e1407ab894603f93c507d3 (127.4.78.67). New cstate: current_term: 1 leader_uuid: "df911a6683e1407ab894603f93c507d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } health_report { overall_health: HEALTHY } } }
W20260924 15:23:10.868081 6354 tablet.cc:2369] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:10.872511 6307 raft_consensus.cc:1261] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Refusing update from remote peer df911a6683e1407ab894603f93c507d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:10.872802 6666 consensus_queue.cc:1035] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:10.873564 6447 raft_consensus.cc:1261] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Refusing update from remote peer df911a6683e1407ab894603f93c507d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:10.873801 6698 consensus_queue.cc:1035] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:23:10.878119 6684 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 15:23:12.395952 6150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:12.396042 6150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:12.396916 6150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: Sending DeleteTablet for 3 replicas of tablet 57f9afab13b949cfbb7cbd8a796a2411
I20260924 15:23:12.397166 6287 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f9afab13b949cfbb7cbd8a796a2411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:12 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:32852
I20260924 15:23:12.397203 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f9afab13b949cfbb7cbd8a796a2411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:12 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:40256
I20260924 15:23:12.397217 6427 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f9afab13b949cfbb7cbd8a796a2411 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:12 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:52466
I20260924 15:23:12.397390 6735 tablet_replica.cc:330] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: stopping tablet replica
I20260924 15:23:12.397450 6735 raft_consensus.cc:2229] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:12.397492 6735 raft_consensus.cc:2258] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:12.397665 6736 tablet_replica.cc:330] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: stopping tablet replica
I20260924 15:23:12.397727 6736 raft_consensus.cc:2229] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:12.397737 6735 ts_tablet_manager.cc:1905] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:12.397787 6736 raft_consensus.cc:2258] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:12.399319 6735 ts_tablet_manager.cc:1918] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:23:12.399394 6735 log.cc:1196] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/57f9afab13b949cfbb7cbd8a796a2411
I20260924 15:23:12.399612 6735 ts_tablet_manager.cc:1939] T 57f9afab13b949cfbb7cbd8a796a2411 P 0e5f676263a349b5a6e99451b1490cb1: Deleting consensus metadata
I20260924 15:23:12.399724 6736 ts_tablet_manager.cc:1905] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:12.399942 6137 catalog_manager.cc:4841] TS 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477): tablet 57f9afab13b949cfbb7cbd8a796a2411 (table test-table [id=4e2cde53a50a4ffdb0bfe473f55d7555]) successfully deleted
I20260924 15:23:12.400233 6734 tablet_replica.cc:330] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: stopping tablet replica
I20260924 15:23:12.400298 6734 raft_consensus.cc:2229] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:12.400329 6734 raft_consensus.cc:2258] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:12.400548 6734 ts_tablet_manager.cc:1905] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:12.401422 6736 ts_tablet_manager.cc:1918] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:23:12.401487 6736 log.cc:1196] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/57f9afab13b949cfbb7cbd8a796a2411
I20260924 15:23:12.401734 6736 ts_tablet_manager.cc:1939] T 57f9afab13b949cfbb7cbd8a796a2411 P df911a6683e1407ab894603f93c507d3: Deleting consensus metadata
I20260924 15:23:12.402073 6137 catalog_manager.cc:4841] TS df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827): tablet 57f9afab13b949cfbb7cbd8a796a2411 (table test-table [id=4e2cde53a50a4ffdb0bfe473f55d7555]) successfully deleted
I20260924 15:23:12.402302 6734 ts_tablet_manager.cc:1918] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:23:12.402349 6734 log.cc:1196] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/57f9afab13b949cfbb7cbd8a796a2411
I20260924 15:23:12.402549 6734 ts_tablet_manager.cc:1939] T 57f9afab13b949cfbb7cbd8a796a2411 P 23bfbc8f16ad4e6590718543119cdf53: Deleting consensus metadata
I20260924 15:23:12.402864 6136 catalog_manager.cc:4841] TS 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709): tablet 57f9afab13b949cfbb7cbd8a796a2411 (table test-table [id=4e2cde53a50a4ffdb0bfe473f55d7555]) successfully deleted
Sep 24 15:23:12 dist-test-slave-l766 krb5kdc[6105](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:23:12 dist-test-slave-l766 krb5kdc[6105](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263392, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:12.422608 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:12.415247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.4.78.126:38397)
0924 15:23:12.415397 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:12.415400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:12.415444 (+ 44us) server_negotiation.cc:407] Connection header received
0924 15:23:12.415530 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:12.415531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:12.415544 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:12.415629 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:12.416324 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:12.416850 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:12.417590 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:12.417865 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:12.417947 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:12.420128 (+ 2181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:12.420129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:12.420130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:12.420138 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:12.421412 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:12.421745 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:12.421746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:12.421747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:12.421857 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:12.422046 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:12.422047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:12.422047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:12.422161 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:12.422274 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:12.422401 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 15:23:12.422502 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260924 15:23:12.424016 6150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:12.424197 6150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:12.425518 6287 tablet_service.cc:1467] Processing CreateTablet for tablet 7e971b0683b6456a842a3cbaf15735f8 (DEFAULT_TABLE table=test-table [id=2976558f5e93446db1b18edd308b6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:12.425519 6427 tablet_service.cc:1467] Processing CreateTablet for tablet 7e971b0683b6456a842a3cbaf15735f8 (DEFAULT_TABLE table=test-table [id=2976558f5e93446db1b18edd308b6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:12.425545 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 7e971b0683b6456a842a3cbaf15735f8 (DEFAULT_TABLE table=test-table [id=2976558f5e93446db1b18edd308b6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:12.425644 6287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e971b0683b6456a842a3cbaf15735f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:12.425654 6427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e971b0683b6456a842a3cbaf15735f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:12.425688 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e971b0683b6456a842a3cbaf15735f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:12.427115 6745 tablet_bootstrap.cc:492] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: Bootstrap starting.
I20260924 15:23:12.427124 6746 tablet_bootstrap.cc:492] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: Bootstrap starting.
I20260924 15:23:12.427508 6745 tablet_bootstrap.cc:654] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:12.427508 6746 tablet_bootstrap.cc:654] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:12.427845 6747 tablet_bootstrap.cc:492] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: Bootstrap starting.
I20260924 15:23:12.428182 6746 tablet_bootstrap.cc:492] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: No bootstrap required, opened a new log
I20260924 15:23:12.428176 6747 tablet_bootstrap.cc:654] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:12.428243 6746 ts_tablet_manager.cc:1397] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:12.428373 6745 tablet_bootstrap.cc:492] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: No bootstrap required, opened a new log
I20260924 15:23:12.428427 6745 ts_tablet_manager.cc:1397] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:12.428366 6746 raft_consensus.cc:348] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.428464 6746 raft_consensus.cc:374] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:12.428499 6746 raft_consensus.cc:729] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5f676263a349b5a6e99451b1490cb1, State: Initialized, Role: FOLLOWER
I20260924 15:23:12.428540 6745 raft_consensus.cc:348] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.428573 6746 consensus_queue.cc:260] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.428613 6745 raft_consensus.cc:374] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:12.428637 6745 raft_consensus.cc:729] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df911a6683e1407ab894603f93c507d3, State: Initialized, Role: FOLLOWER
I20260924 15:23:12.428686 6745 consensus_queue.cc:260] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.428769 6746 ts_tablet_manager.cc:1428] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:12.429179 6747 tablet_bootstrap.cc:492] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: No bootstrap required, opened a new log
I20260924 15:23:12.429231 6747 ts_tablet_manager.cc:1397] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:12.429245 6745 ts_tablet_manager.cc:1428] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:12.429333 6747 raft_consensus.cc:348] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.429407 6747 raft_consensus.cc:374] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:12.429425 6747 raft_consensus.cc:729] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bfbc8f16ad4e6590718543119cdf53, State: Initialized, Role: FOLLOWER
I20260924 15:23:12.429467 6747 consensus_queue.cc:260] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.429567 6747 ts_tablet_manager.cc:1428] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 15:23:12.432878 6634 tablet.cc:2369] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:12.478896 6720 raft_consensus.cc:482] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:12.478993 6720 raft_consensus.cc:504] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.479194 6720 leader_election.cc:288] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477), df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827)
I20260924 15:23:12.479420 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e971b0683b6456a842a3cbaf15735f8" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1" is_pre_election: true
I20260924 15:23:12.479427 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e971b0683b6456a842a3cbaf15735f8" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df911a6683e1407ab894603f93c507d3" is_pre_election: true
I20260924 15:23:12.479537 6446 raft_consensus.cc:2454] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 0.
I20260924 15:23:12.479578 6587 raft_consensus.cc:2454] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 0.
I20260924 15:23:12.479724 6241 leader_election.cc:302] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23bfbc8f16ad4e6590718543119cdf53, df911a6683e1407ab894603f93c507d3; no voters:
I20260924 15:23:12.479854 6720 raft_consensus.cc:2781] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:12.479897 6720 raft_consensus.cc:482] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:12.479921 6720 raft_consensus.cc:3037] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:12.480628 6720 raft_consensus.cc:504] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.480754 6720 leader_election.cc:288] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 election: Requested vote from peers 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477), df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827)
I20260924 15:23:12.480943 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e971b0683b6456a842a3cbaf15735f8" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df911a6683e1407ab894603f93c507d3"
I20260924 15:23:12.480942 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e971b0683b6456a842a3cbaf15735f8" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1"
I20260924 15:23:12.481006 6446 raft_consensus.cc:3037] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:12.481027 6587 raft_consensus.cc:3037] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:12.481650 6587 raft_consensus.cc:2454] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 1.
I20260924 15:23:12.481659 6446 raft_consensus.cc:2454] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 1.
I20260924 15:23:12.481890 6241 leader_election.cc:302] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f676263a349b5a6e99451b1490cb1, 23bfbc8f16ad4e6590718543119cdf53; no voters:
I20260924 15:23:12.481983 6720 raft_consensus.cc:2781] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:12.482045 6720 raft_consensus.cc:686] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 LEADER]: Becoming Leader. State: Replica: 23bfbc8f16ad4e6590718543119cdf53, State: Running, Role: LEADER
I20260924 15:23:12.482087 6720 consensus_queue.cc:237] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.482693 6150 catalog_manager.cc:5495] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } health_report { overall_health: UNKNOWN } } }
I20260924 15:23:12.489550 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:12.487119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39166 (local address 127.4.78.65:36709)
0924 15:23:12.487273 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:12.487275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:12.487282 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:12.487302 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:12.487303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:12.487319 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:12.487390 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:12.487497 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:12.488036 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:12.488822 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:12.489053 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:12.489145 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:12.489300 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:12.489335 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:12.489389 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:12.489446 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 15:23:12.489469 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260924 15:23:12.493549 6754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:12.491470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:40089 (local address 127.4.78.65:36709)
0924 15:23:12.491524 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:12.491526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:12.491610 (+ 84us) server_negotiation.cc:407] Connection header received
0924 15:23:12.491660 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:12.491660 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:12.491669 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:12.491711 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:12.491801 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:12.492273 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:12.492984 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:12.493181 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:12.493228 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:12.493345 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:12.493366 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:12.493403 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:12.493451 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 15:23:12.493483 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":38}
I20260924 15:23:12.493590 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:12.491421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36709 (local address 127.4.78.65:40089)
0924 15:23:12.491550 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:12.491559 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:12.491617 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:12.491720 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:12.491720 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:12.491721 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:12.491765 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:12.491766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:12.492331 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:12.492331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:12.492892 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:12.492894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:12.493244 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:12.493244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:12.493284 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:12.493287 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:12.493415 (+ 128us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:12.493416 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:23:12.493449 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 15:23:12.493497 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":3,"thread_start_us":93,"threads_started":1}
I20260924 15:23:12.495426 6446 raft_consensus.cc:1261] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Refusing update from remote peer 23bfbc8f16ad4e6590718543119cdf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:12.495436 6587 raft_consensus.cc:1261] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Refusing update from remote peer 23bfbc8f16ad4e6590718543119cdf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:12.495644 6720 consensus_queue.cc:1035] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:12.495757 6715 consensus_queue.cc:1035] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:12.506099 6150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:12.506225 6150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:12.506839 6150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: Sending DeleteTablet for 3 replicas of tablet 7e971b0683b6456a842a3cbaf15735f8
I20260924 15:23:12.507072 6427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e971b0683b6456a842a3cbaf15735f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:12 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:52466
I20260924 15:23:12.507176 6735 tablet_replica.cc:330] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: stopping tablet replica
I20260924 15:23:12.507165 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e971b0683b6456a842a3cbaf15735f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:12 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:40256
I20260924 15:23:12.507222 6735 raft_consensus.cc:2229] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:12.507243 6736 tablet_replica.cc:330] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: stopping tablet replica
I20260924 15:23:12.507261 6735 raft_consensus.cc:2258] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:12.507301 6736 raft_consensus.cc:2229] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:12.507334 6736 raft_consensus.cc:2258] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:12.507513 6287 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e971b0683b6456a842a3cbaf15735f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:12 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:32852
I20260924 15:23:12.507594 6734 tablet_replica.cc:330] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: stopping tablet replica
I20260924 15:23:12.507647 6734 raft_consensus.cc:2229] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:12.507726 6734 raft_consensus.cc:2258] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:12.507769 6735 ts_tablet_manager.cc:1905] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:12.507928 6150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:12.508042 6150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:12.508055 6734 ts_tablet_manager.cc:1905] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:12.509187 6567 tablet_service.cc:1467] Processing CreateTablet for tablet 8b99a64e4451488ab5db3f4566aff123 (DEFAULT_TABLE table=test-table [id=4ab1b14310804f449fe735e12d43252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:12.509306 6567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b99a64e4451488ab5db3f4566aff123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:12.509402 6735 ts_tablet_manager.cc:1918] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:12.509450 6735 log.cc:1196] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/7e971b0683b6456a842a3cbaf15735f8
I20260924 15:23:12.509488 6734 ts_tablet_manager.cc:1918] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:12.509536 6734 log.cc:1196] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/7e971b0683b6456a842a3cbaf15735f8
I20260924 15:23:12.509660 6735 ts_tablet_manager.cc:1939] T 7e971b0683b6456a842a3cbaf15735f8 P 0e5f676263a349b5a6e99451b1490cb1: Deleting consensus metadata
I20260924 15:23:12.509749 6734 ts_tablet_manager.cc:1939] T 7e971b0683b6456a842a3cbaf15735f8 P 23bfbc8f16ad4e6590718543119cdf53: Deleting consensus metadata
I20260924 15:23:12.510298 6137 catalog_manager.cc:4841] TS 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477): tablet 7e971b0683b6456a842a3cbaf15735f8 (table test-table [id=2976558f5e93446db1b18edd308b6bdc]) successfully deleted
I20260924 15:23:12.510635 6745 tablet_bootstrap.cc:492] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: Bootstrap starting.
I20260924 15:23:12.510728 6427 tablet_service.cc:1467] Processing CreateTablet for tablet 8b99a64e4451488ab5db3f4566aff123 (DEFAULT_TABLE table=test-table [id=4ab1b14310804f449fe735e12d43252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:12.510818 6427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b99a64e4451488ab5db3f4566aff123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:12.510955 6745 tablet_bootstrap.cc:654] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:12.511540 6745 tablet_bootstrap.cc:492] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: No bootstrap required, opened a new log
I20260924 15:23:12.511595 6745 ts_tablet_manager.cc:1397] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:12.511685 6745 raft_consensus.cc:348] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.511749 6745 raft_consensus.cc:374] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:12.511775 6745 raft_consensus.cc:729] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df911a6683e1407ab894603f93c507d3, State: Initialized, Role: FOLLOWER
I20260924 15:23:12.511829 6745 consensus_queue.cc:260] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.511948 6745 ts_tablet_manager.cc:1428] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:12.512018 6746 tablet_bootstrap.cc:492] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: Bootstrap starting.
I20260924 15:23:12.512367 6746 tablet_bootstrap.cc:654] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:12.512858 6136 catalog_manager.cc:4841] TS 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709): tablet 7e971b0683b6456a842a3cbaf15735f8 (table test-table [id=2976558f5e93446db1b18edd308b6bdc]) successfully deleted
I20260924 15:23:12.512966 6287 tablet_service.cc:1467] Processing CreateTablet for tablet 8b99a64e4451488ab5db3f4566aff123 (DEFAULT_TABLE table=test-table [id=4ab1b14310804f449fe735e12d43252e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:12.513041 6287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b99a64e4451488ab5db3f4566aff123. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:12.513849 6747 tablet_bootstrap.cc:492] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53: Bootstrap starting.
I20260924 15:23:12.514117 6746 tablet_bootstrap.cc:492] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: No bootstrap required, opened a new log
I20260924 15:23:12.514149 6746 ts_tablet_manager.cc:1397] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:12.514164 6747 tablet_bootstrap.cc:654] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:12.514247 6746 raft_consensus.cc:348] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.514297 6746 raft_consensus.cc:374] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:12.514309 6746 raft_consensus.cc:729] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5f676263a349b5a6e99451b1490cb1, State: Initialized, Role: FOLLOWER
I20260924 15:23:12.514367 6746 consensus_queue.cc:260] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.514453 6746 ts_tablet_manager.cc:1428] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:12.514798 6736 ts_tablet_manager.cc:1905] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:12.515082 6747 tablet_bootstrap.cc:492] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53: No bootstrap required, opened a new log
I20260924 15:23:12.515128 6747 ts_tablet_manager.cc:1397] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:12.515244 6747 raft_consensus.cc:348] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.515298 6747 raft_consensus.cc:374] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:12.515318 6747 raft_consensus.cc:729] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23bfbc8f16ad4e6590718543119cdf53, State: Initialized, Role: FOLLOWER
I20260924 15:23:12.515362 6747 consensus_queue.cc:260] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.515455 6747 ts_tablet_manager.cc:1428] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:12.516332 6736 ts_tablet_manager.cc:1918] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:12.516391 6736 log.cc:1196] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/7e971b0683b6456a842a3cbaf15735f8
I20260924 15:23:12.516569 6736 ts_tablet_manager.cc:1939] T 7e971b0683b6456a842a3cbaf15735f8 P df911a6683e1407ab894603f93c507d3: Deleting consensus metadata
I20260924 15:23:12.516924 6137 catalog_manager.cc:4841] TS df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827): tablet 7e971b0683b6456a842a3cbaf15735f8 (table test-table [id=2976558f5e93446db1b18edd308b6bdc]) successfully deleted
I20260924 15:23:12.589900 6715 raft_consensus.cc:482] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:12.590008 6715 raft_consensus.cc:504] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.590189 6715 leader_election.cc:288] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477), df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827)
I20260924 15:23:12.590411 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b99a64e4451488ab5db3f4566aff123" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1" is_pre_election: true
I20260924 15:23:12.590401 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b99a64e4451488ab5db3f4566aff123" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df911a6683e1407ab894603f93c507d3" is_pre_election: true
I20260924 15:23:12.590507 6587 raft_consensus.cc:2454] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 0.
I20260924 15:23:12.590519 6446 raft_consensus.cc:2454] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 0.
I20260924 15:23:12.590662 6241 leader_election.cc:302] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f676263a349b5a6e99451b1490cb1, 23bfbc8f16ad4e6590718543119cdf53; no voters:
I20260924 15:23:12.590822 6715 raft_consensus.cc:2781] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:12.590875 6715 raft_consensus.cc:482] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:12.590889 6715 raft_consensus.cc:3037] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:12.591593 6715 raft_consensus.cc:504] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.591717 6715 leader_election.cc:288] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 election: Requested vote from peers 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477), df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827)
I20260924 15:23:12.591871 6446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b99a64e4451488ab5db3f4566aff123" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e5f676263a349b5a6e99451b1490cb1"
I20260924 15:23:12.591881 6587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b99a64e4451488ab5db3f4566aff123" candidate_uuid: "23bfbc8f16ad4e6590718543119cdf53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df911a6683e1407ab894603f93c507d3"
I20260924 15:23:12.591951 6587 raft_consensus.cc:3037] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:12.591948 6446 raft_consensus.cc:3037] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:12.592536 6446 raft_consensus.cc:2454] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 1.
I20260924 15:23:12.592715 6241 leader_election.cc:302] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e5f676263a349b5a6e99451b1490cb1, 23bfbc8f16ad4e6590718543119cdf53; no voters:
I20260924 15:23:12.592777 6587 raft_consensus.cc:2454] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23bfbc8f16ad4e6590718543119cdf53 in term 1.
I20260924 15:23:12.592829 6715 raft_consensus.cc:2781] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:12.592877 6715 raft_consensus.cc:686] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 LEADER]: Becoming Leader. State: Replica: 23bfbc8f16ad4e6590718543119cdf53, State: Running, Role: LEADER
I20260924 15:23:12.592932 6715 consensus_queue.cc:237] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } }
I20260924 15:23:12.593541 6149 catalog_manager.cc:5495] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "23bfbc8f16ad4e6590718543119cdf53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23bfbc8f16ad4e6590718543119cdf53" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36709 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 } health_report { overall_health: UNKNOWN } } }
I20260924 15:23:12.611661 6446 raft_consensus.cc:1261] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Refusing update from remote peer 23bfbc8f16ad4e6590718543119cdf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:12.611661 6587 raft_consensus.cc:1261] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Refusing update from remote peer 23bfbc8f16ad4e6590718543119cdf53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:12.611913 6715 consensus_queue.cc:1035] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df911a6683e1407ab894603f93c507d3" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:12.612044 6720 consensus_queue.cc:1035] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e5f676263a349b5a6e99451b1490cb1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:23:12.616122 6713 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 15:23:14.133170 6150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:14.133278 6150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:14.134099 6150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6ada7bca1b634b83a9402df2d5c0258c: Sending DeleteTablet for 3 replicas of tablet 8b99a64e4451488ab5db3f4566aff123
I20260924 15:23:14.134335 6287 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b99a64e4451488ab5db3f4566aff123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:14 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:32852
I20260924 15:23:14.134526 6567 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b99a64e4451488ab5db3f4566aff123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:14 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:40256
I20260924 15:23:14.134533 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 6222
I20260924 15:23:14.134680 6786 tablet_replica.cc:330] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53: stopping tablet replica
I20260924 15:23:14.134708 6787 tablet_replica.cc:330] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: stopping tablet replica
I20260924 15:23:14.134756 6787 raft_consensus.cc:2229] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:14.134793 6787 raft_consensus.cc:2258] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:14.134877 6786 raft_consensus.cc:2229] T 8b99a64e4451488ab5db3f4566aff123 P 23bfbc8f16ad4e6590718543119cdf53 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:14.135025 6427 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b99a64e4451488ab5db3f4566aff123 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:14 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:52466
I20260924 15:23:14.135134 6787 ts_tablet_manager.cc:1905] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:14.135246 6788 tablet_replica.cc:330] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: stopping tablet replica
I20260924 15:23:14.135341 6788 raft_consensus.cc:2229] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:14.135421 6788 raft_consensus.cc:2258] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:14.135696 6788 ts_tablet_manager.cc:1905] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:14.137342 6787 ts_tablet_manager.cc:1918] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:23:14.137426 6787 log.cc:1196] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/8b99a64e4451488ab5db3f4566aff123
I20260924 15:23:14.137657 6787 ts_tablet_manager.cc:1939] T 8b99a64e4451488ab5db3f4566aff123 P df911a6683e1407ab894603f93c507d3: Deleting consensus metadata
I20260924 15:23:14.137804 6788 ts_tablet_manager.cc:1918] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:23:14.137899 6788 log.cc:1196] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/8b99a64e4451488ab5db3f4566aff123
I20260924 15:23:14.138002 6137 catalog_manager.cc:4841] TS df911a6683e1407ab894603f93c507d3 (127.4.78.67:39827): tablet 8b99a64e4451488ab5db3f4566aff123 (table test-table [id=4ab1b14310804f449fe735e12d43252e]) successfully deleted
I20260924 15:23:14.138178 6788 ts_tablet_manager.cc:1939] T 8b99a64e4451488ab5db3f4566aff123 P 0e5f676263a349b5a6e99451b1490cb1: Deleting consensus metadata
I20260924 15:23:14.138481 6137 catalog_manager.cc:4841] TS 0e5f676263a349b5a6e99451b1490cb1 (127.4.78.66:43477): tablet 8b99a64e4451488ab5db3f4566aff123 (table test-table [id=4ab1b14310804f449fe735e12d43252e]) successfully deleted
W20260924 15:23:14.142416 6136 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:36709 (error 108)
I20260924 15:23:14.142976 6790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:14.142790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36709 (local address 127.0.0.1:39172)
0924 15:23:14.142905 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:14.142945 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:36709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260924 15:23:14.142975 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 6362
W20260924 15:23:14.143085 6136 catalog_manager.cc:4584] TS 23bfbc8f16ad4e6590718543119cdf53 (127.4.78.65:36709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b99a64e4451488ab5db3f4566aff123: Network error: Client connection negotiation failed: client connection to 127.4.78.65:36709: connect: Connection refused (error 111)
I20260924 15:23:14.147867 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 6502
I20260924 15:23:14.153532 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 6118
2026-09-24T15:23:14Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5251 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:14 dist-test-slave-l766 krb5kdc[6793](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:14 dist-test-slave-l766 krb5kdc[6793](info): set up 2 sockets
Sep 24 15:23:14 dist-test-slave-l766 krb5kdc[6793](info): commencing operation
krb5kdc: starting...
W20260924 15:23:15.180588 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:15.296718 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:44575
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:44575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:15.376303 6806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:15.376488 6806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:15.376510 6806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:15.377579 6806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:15.377614 6806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:15.377628 6806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:15.377640 6806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:15.377652 6806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:15.379081 6806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:44575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:44575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:15.379266 6806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:15.379436 6806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:15.383992 6812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:15.383968 6814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:15.383976 6811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:15.384249 6806 server_base.cc:1034] running on GCE node
I20260924 15:23:15.384482 6806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:15.384696 6806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:15.385866 6806 hybrid_clock.cc:648] HybridClock initialized: now 1790263395385845 us; error 30 us; skew 500 ppm
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:15.388212 6806 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:15.388576 6806 webserver.cc:460] Webserver started at http://127.4.78.126:41161/ using document root <none> and password file <none>
I20260924 15:23:15.388762 6806 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:15.388803 6806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:15.388886 6806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:15.389917 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "6612c0eeee2a4517a1074bec9a22ca8a"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "573b4268e6c7d574a26fef9a1c310404"
server_key_iv: "adb83f6d529a3eb3f3443fc87cb67310"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.390246 6806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "6612c0eeee2a4517a1074bec9a22ca8a"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "573b4268e6c7d574a26fef9a1c310404"
server_key_iv: "adb83f6d529a3eb3f3443fc87cb67310"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.391541 6806 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:15.392190 6821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.392346 6806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:15.392444 6806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "6612c0eeee2a4517a1074bec9a22ca8a"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "573b4268e6c7d574a26fef9a1c310404"
server_key_iv: "adb83f6d529a3eb3f3443fc87cb67310"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.392515 6806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:15.405192 6806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:15.406877 6806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:15.407028 6806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:15.410934 6806 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:44575
I20260924 15:23:15.410959 6873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:44575 every 8 connection(s)
I20260924 15:23:15.411307 6806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:15.412108 6874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:15.414227 6874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a: Bootstrap starting.
I20260924 15:23:15.414714 6874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:15.414940 6874 log.cc:824] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:15.415524 6874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a: No bootstrap required, opened a new log
I20260924 15:23:15.417028 6874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c0eeee2a4517a1074bec9a22ca8a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44575 } }
I20260924 15:23:15.417160 6874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:15.417184 6874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6612c0eeee2a4517a1074bec9a22ca8a, State: Initialized, Role: FOLLOWER
I20260924 15:23:15.417321 6874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c0eeee2a4517a1074bec9a22ca8a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44575 } }
I20260924 15:23:15.417395 6874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:15.417418 6874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:15.417449 6874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:15.417959 6874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c0eeee2a4517a1074bec9a22ca8a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44575 } }
I20260924 15:23:15.418042 6874 leader_election.cc:302] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6612c0eeee2a4517a1074bec9a22ca8a; no voters:
I20260924 15:23:15.418167 6874 leader_election.cc:288] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:15.418226 6877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:15.418409 6874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:15.418589 6877 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [term 1 LEADER]: Becoming Leader. State: Replica: 6612c0eeee2a4517a1074bec9a22ca8a, State: Running, Role: LEADER
I20260924 15:23:15.418730 6877 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c0eeee2a4517a1074bec9a22ca8a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44575 } }
I20260924 15:23:15.419099 6878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6612c0eeee2a4517a1074bec9a22ca8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c0eeee2a4517a1074bec9a22ca8a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44575 } } }
I20260924 15:23:15.419175 6878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:15.419122 6879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6612c0eeee2a4517a1074bec9a22ca8a. Latest consensus state: current_term: 1 leader_uuid: "6612c0eeee2a4517a1074bec9a22ca8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6612c0eeee2a4517a1074bec9a22ca8a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44575 } } }
I20260924 15:23:15.419236 6879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:15.419411 6886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:15.419957 6886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:15.421089 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 6806
I20260924 15:23:15.421176 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:15.421346 4409 external_mini_cluster.cc:1428] Setting key 7d116842ccedff5e8845c5b0361b2e2e
I20260924 15:23:15.421439 6886 catalog_manager.cc:1347] Generated new cluster ID: fbc7d38dc9be43fba2f65262077c9cc8
I20260924 15:23:15.421480 6886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:15.432660 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:15.421876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.4.78.126:44575)
0924 15:23:15.422011 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:15.422019 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:15.422047 (+ 28us) server_negotiation.cc:407] Connection header received
0924 15:23:15.422161 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:15.422168 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:15.422216 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:15.422365 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:15.423117 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.424501 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.425546 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.426189 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.426286 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:15.429026 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:15.429030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:15.429031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:15.429042 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:15.431063 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:15.431444 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:15.431451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:15.431452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:15.431563 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:15.431756 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:15.431757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:15.431757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:15.432054 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:15.432214 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:15.432288 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 15:23:15.432424 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260924 15:23:15.458623 6886 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:15.459650 6886 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:15.462109 6886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6612c0eeee2a4517a1074bec9a22ca8a: Generated new TSK 0
I20260924 15:23:15.462291 6886 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:15.494865 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:44575
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:15.574894 6900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:15.575131 6900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:15.575168 6900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:15.576323 6900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:15.576385 6900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:15.576428 6900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:15.577872 6900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:15.578121 6900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:15.578341 6900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:15.583257 6906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:15.583288 6905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:15.583354 6900 server_base.cc:1034] running on GCE node
W20260924 15:23:15.583264 6908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:15.583686 6900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:15.583920 6900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:15.585054 6900 hybrid_clock.cc:648] HybridClock initialized: now 1790263395585039 us; error 35 us; skew 500 ppm
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:15.587548 6900 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:15.587950 6900 webserver.cc:460] Webserver started at http://127.4.78.65:45471/ using document root <none> and password file <none>
I20260924 15:23:15.588162 6900 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:15.588227 6900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:15.588371 6900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:15.589367 6900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "409eb2f05262489cb929e566e5fb52b0"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "b9e1891baae054f31601f62eaff0d95a"
server_key_iv: "ab59c3dc6e644b940bfd786f69d6e17a"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.589715 6900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "409eb2f05262489cb929e566e5fb52b0"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "b9e1891baae054f31601f62eaff0d95a"
server_key_iv: "ab59c3dc6e644b940bfd786f69d6e17a"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.591041 6900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:15.591724 6915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.591886 6900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:15.591985 6900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "409eb2f05262489cb929e566e5fb52b0"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "b9e1891baae054f31601f62eaff0d95a"
server_key_iv: "ab59c3dc6e644b940bfd786f69d6e17a"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.592059 6900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:15.610541 6900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:15.612856 6900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:15.613018 6900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:15.613250 6900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:15.613574 6900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:15.613623 6900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.613663 6900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:15.613691 6900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.619896 6900 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:40637
I20260924 15:23:15.619946 7028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:40637 every 8 connection(s)
I20260924 15:23:15.620275 6900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:15.629091 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:15.620931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:41141 (local address 127.4.78.126:44575)
0924 15:23:15.620996 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:15.620998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:15.621192 (+ 194us) server_negotiation.cc:407] Connection header received
0924 15:23:15.621654 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:15.621655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:15.621667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:15.621729 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:15.622746 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.623368 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.624331 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.624522 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.624572 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:15.626586 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:15.626588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:15.626588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:15.626596 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:15.627966 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:15.628290 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:15.628291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:15.628291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:15.628340 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:15.628536 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:15.628537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:15.628537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:15.628629 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:15.628694 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:15.628937 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 15:23:15.629002 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:23:15.629251 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:15.620903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:44575 (local address 127.4.78.65:41141)
0924 15:23:15.621065 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:15.621106 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:15.621574 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:15.621784 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:15.621787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:15.622393 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:15.622574 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:15.622579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.623446 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.623446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:15.624259 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:15.624261 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.624589 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.624590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:15.624692 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:15.625129 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:15.625136 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:15.626495 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:15.628031 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:15.628035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:15.628037 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:15.628227 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:15.628391 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:15.628392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:15.628392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:15.628489 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:15.628697 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:15.628701 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:15.628806 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 15:23:15.628916 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:23:15.629036 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260924 15:23:15.629480 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 6900
I20260924 15:23:15.629566 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:15.629756 4409 external_mini_cluster.cc:1428] Setting key 93cba33180ca7ed93c2bdc0485daf370
I20260924 15:23:15.629796 7029 heartbeater.cc:347] Connected to a master server at 127.4.78.126:44575
I20260924 15:23:15.629902 7029 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:15.630079 7029 heartbeater.cc:511] Master 127.4.78.126:44575 requested a full tablet report, sending...
I20260924 15:23:15.630589 6838 ts_manager.cc:194] Registered new tserver with Master: 409eb2f05262489cb929e566e5fb52b0 (127.4.78.65:40637)
I20260924 15:23:15.631179 6838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:41141
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:15.653043 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:44575
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:15.733340 7034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:15.733539 7034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:15.733561 7034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:15.734738 7034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:15.734781 7034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:15.734828 7034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:15.736243 7034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:15.736485 7034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:15.736707 7034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:15.741513 7039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:15.741532 7040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:15.741510 7042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:15.741595 7034 server_base.cc:1034] running on GCE node
I20260924 15:23:15.741957 7034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:15.742214 7034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:15.743373 7034 hybrid_clock.cc:648] HybridClock initialized: now 1790263395743366 us; error 34 us; skew 500 ppm
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:15.745739 7034 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:15.746156 7034 webserver.cc:460] Webserver started at http://127.4.78.66:43505/ using document root <none> and password file <none>
I20260924 15:23:15.746345 7034 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:15.746384 7034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:15.746472 7034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:15.747452 7034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "179c341093114b9493f20b43689193eb"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "1243dbeaa8ae3eb50ace892937d30646"
server_key_iv: "319a2aa3aa006f33062416b9e9a7df2c"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.747749 7034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "179c341093114b9493f20b43689193eb"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "1243dbeaa8ae3eb50ace892937d30646"
server_key_iv: "319a2aa3aa006f33062416b9e9a7df2c"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.748940 7034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:15.749572 7049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.749718 7034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:15.749801 7034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "179c341093114b9493f20b43689193eb"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "1243dbeaa8ae3eb50ace892937d30646"
server_key_iv: "319a2aa3aa006f33062416b9e9a7df2c"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.749907 7034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:15.763348 7034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:15.765539 7034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:15.765647 7034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:15.765875 7034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:15.766145 7034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:15.766176 7034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.766197 7034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:15.766211 7034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.771677 7034 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:43489
I20260924 15:23:15.771751 7162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:43489 every 8 connection(s)
I20260924 15:23:15.772042 7034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:15.778004 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 7034
I20260924 15:23:15.778110 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:15.778293 4409 external_mini_cluster.cc:1428] Setting key 3869f1c08284149f20e4a3031df92c6c
I20260924 15:23:15.781268 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:15.772744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:45953 (local address 127.4.78.126:44575)
0924 15:23:15.772826 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:15.772829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:15.773006 (+ 177us) server_negotiation.cc:407] Connection header received
0924 15:23:15.773470 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:15.773471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:15.773482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:15.773533 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:15.774500 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.775042 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.775979 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.776166 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.776216 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:15.778350 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:15.778352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:15.778353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:15.778361 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:15.779904 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:15.780307 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:15.780309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:15.780309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:15.780368 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:15.780630 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:15.780631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:15.780631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:15.780745 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:15.780817 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:15.781093 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 15:23:15.781181 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:23:15.781440 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:15.772716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:44575 (local address 127.4.78.66:45953)
0924 15:23:15.772885 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:15.772919 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:15.773388 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:15.773584 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:15.773587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:15.774213 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:15.774408 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:15.774413 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.775126 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.775127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:15.775910 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:15.775911 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.776233 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.776234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:15.776343 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:15.776834 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:15.776842 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:15.778249 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:15.779993 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:15.780000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:15.780002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:15.780220 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:15.780430 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:15.780431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:15.780432 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:15.780547 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:15.780807 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:15.780814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:15.780954 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 15:23:15.781052 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 15:23:15.781207 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260924 15:23:15.781901 7163 heartbeater.cc:347] Connected to a master server at 127.4.78.126:44575
I20260924 15:23:15.782007 7163 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:15.782177 7163 heartbeater.cc:511] Master 127.4.78.126:44575 requested a full tablet report, sending...
I20260924 15:23:15.782567 6838 ts_manager.cc:194] Registered new tserver with Master: 179c341093114b9493f20b43689193eb (127.4.78.66:43489)
I20260924 15:23:15.782945 6838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:45953
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:15.804149 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:44575
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:15.884254 7168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:15.884442 7168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:15.884464 7168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:15.885494 7168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:15.885525 7168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:15.885550 7168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:15.886893 7168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:44575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:15.887074 7168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:15.887244 7168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:23:15.892067 7168 server_base.cc:1034] running on GCE node
W20260924 15:23:15.892042 7176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:15.892032 7173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:15.892012 7174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:15.892494 7168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:15.892722 7168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:15.893865 7168 hybrid_clock.cc:648] HybridClock initialized: now 1790263395893850 us; error 28 us; skew 500 ppm
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:15.896454 7168 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:15.896890 7168 webserver.cc:460] Webserver started at http://127.4.78.67:33477/ using document root <none> and password file <none>
I20260924 15:23:15.897074 7168 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:15.897114 7168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:15.897198 7168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:15.898188 7168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "cc976524262346268eb1965205488058"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "8341ed1d32945e74de0d661d06aa2740"
server_key_iv: "b94208b1972747fc47296659816616e2"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.898535 7168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "cc976524262346268eb1965205488058"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "8341ed1d32945e74de0d661d06aa2740"
server_key_iv: "b94208b1972747fc47296659816616e2"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.899720 7168 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:15.900358 7183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.900522 7168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:15.900621 7168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "cc976524262346268eb1965205488058"
format_stamp: "Formatted at 2026-09-24 15:23:15 on dist-test-slave-l766"
server_key: "8341ed1d32945e74de0d661d06aa2740"
server_key_iv: "b94208b1972747fc47296659816616e2"
server_key_version: "encryptionkey@0"
I20260924 15:23:15.900691 7168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:15.914691 7168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:15.916879 7168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:15.916989 7168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:15.917147 7168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:15.917410 7168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:15.917446 7168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.917469 7168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:15.917488 7168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:15.923372 7168 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:39567
I20260924 15:23:15.923440 7296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:39567 every 8 connection(s)
I20260924 15:23:15.923758 7168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:15.928704 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 7168
I20260924 15:23:15.928829 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:15.929062 4409 external_mini_cluster.cc:1428] Setting key a96bc73718be745ef4274c372c800d6a
Sep 24 15:23:15 dist-test-slave-l766 krb5kdc[6793](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263395, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:15.932785 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:15.924469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:41249 (local address 127.4.78.126:44575)
0924 15:23:15.924527 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:15.924529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:15.924740 (+ 211us) server_negotiation.cc:407] Connection header received
0924 15:23:15.925215 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:15.925216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:15.925225 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:15.925267 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:15.926239 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.926722 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.927649 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.927823 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.927870 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:15.930098 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:15.930100 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:15.930100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:15.930108 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:15.931472 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:15.931868 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:15.931869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:15.931870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:15.931927 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:15.932176 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:15.932177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:15.932177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:15.932287 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:15.932356 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:15.932619 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 15:23:15.932697 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:23:15.932969 7299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:15.924455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:44575 (local address 127.4.78.67:41249)
0924 15:23:15.924602 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:15.924643 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:15.925107 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:15.925323 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:15.925326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:15.925950 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:15.926153 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:15.926158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.926793 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.926794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:15.927582 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:15.927584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:15.927890 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:15.927890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:15.927988 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:15.928418 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:15.928425 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:15.930001 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:15.931570 (+ 1569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:15.931577 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:15.931578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:15.931796 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:15.931990 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:15.931991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:15.931991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:15.932098 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:15.932348 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:15.932354 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:15.932487 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 15:23:15.932585 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 15:23:15.932745 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260924 15:23:15.933346 7297 heartbeater.cc:347] Connected to a master server at 127.4.78.126:44575
I20260924 15:23:15.933420 7297 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:15.933549 7297 heartbeater.cc:511] Master 127.4.78.126:44575 requested a full tablet report, sending...
I20260924 15:23:15.933862 6838 ts_manager.cc:194] Registered new tserver with Master: cc976524262346268eb1965205488058 (127.4.78.67:39567)
I20260924 15:23:15.934334 6838 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:41249
I20260924 15:23:15.940649 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:23:15.951875 6897 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:15.951005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.4.78.126:44575)
0924 15:23:15.951077 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:15.951079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:15.951179 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:23:15.951252 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:15.951253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:15.951263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:15.951329 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:15.951811 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53142: BlockingRecv error: recv got EOF from 127.0.0.1:53142 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:23:15.952868 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 6900
I20260924 15:23:15.957659 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 7034
I20260924 15:23:15.961796 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 7168
I20260924 15:23:15.966066 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 6806
2026-09-24T15:23:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1811 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:16 dist-test-slave-l766 krb5kdc[7309](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:16 dist-test-slave-l766 krb5kdc[7309](info): set up 2 sockets
Sep 24 15:23:16 dist-test-slave-l766 krb5kdc[7309](info): commencing operation
krb5kdc: starting...
W20260924 15:23:16.992528 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:17 dist-test-slave-l766 krb5kdc[7309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:17.106433 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:43391
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:43391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:17.186290 7322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:17.186472 7322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:17.186493 7322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:17.187670 7322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:17.187711 7322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:17.187726 7322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:17.187772 7322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:17.187795 7322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:17.189294 7322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:43391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:43391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:17.189525 7322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:17.189736 7322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:17.194617 7330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:17.194590 7328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:17.194598 7322 server_base.cc:1034] running on GCE node
W20260924 15:23:17.194598 7327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:17.195061 7322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:17.195348 7322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:17.196488 7322 hybrid_clock.cc:648] HybridClock initialized: now 1790263397196475 us; error 28 us; skew 500 ppm
Sep 24 15:23:17 dist-test-slave-l766 krb5kdc[7309](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263397, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:17.199092 7322 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:17.199559 7322 webserver.cc:460] Webserver started at http://127.4.78.126:34881/ using document root <none> and password file <none>
I20260924 15:23:17.199791 7322 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:17.199837 7322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:17.200003 7322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:17.201056 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "28425c5da2ae4ea59b92ec828250b4ec"
format_stamp: "Formatted at 2026-09-24 15:23:17 on dist-test-slave-l766"
server_key: "9b89bfb9dbf56ef22bddc62ab1f42afa"
server_key_iv: "cab5c477f2f38023585d809eb8b57de0"
server_key_version: "encryptionkey@0"
I20260924 15:23:17.201400 7322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "28425c5da2ae4ea59b92ec828250b4ec"
format_stamp: "Formatted at 2026-09-24 15:23:17 on dist-test-slave-l766"
server_key: "9b89bfb9dbf56ef22bddc62ab1f42afa"
server_key_iv: "cab5c477f2f38023585d809eb8b57de0"
server_key_version: "encryptionkey@0"
I20260924 15:23:17.202736 7322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:17.203394 7337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:17.203557 7322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:17.203657 7322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "28425c5da2ae4ea59b92ec828250b4ec"
format_stamp: "Formatted at 2026-09-24 15:23:17 on dist-test-slave-l766"
server_key: "9b89bfb9dbf56ef22bddc62ab1f42afa"
server_key_iv: "cab5c477f2f38023585d809eb8b57de0"
server_key_version: "encryptionkey@0"
I20260924 15:23:17.203728 7322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:17.212534 7322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:17.214982 7322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:17.215135 7322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:17.218910 7322 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:43391
I20260924 15:23:17.219362 7322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:17.220773 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 7322
I20260924 15:23:17.220854 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:17.221014 4409 external_mini_cluster.cc:1428] Setting key b1a39593f1df44d801f7ec009bde00d0
I20260924 15:23:17.221938 7390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:43391 every 8 connection(s)
I20260924 15:23:17.222887 7391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:17.225967 7391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec: Bootstrap starting.
I20260924 15:23:17.226715 7391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:17.227097 7391 log.cc:824] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:17.228012 7391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec: No bootstrap required, opened a new log
Sep 24 15:23:17 dist-test-slave-l766 krb5kdc[7309](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:17.230093 7391 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425c5da2ae4ea59b92ec828250b4ec" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43391 } }
I20260924 15:23:17.230278 7391 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:17.230317 7391 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28425c5da2ae4ea59b92ec828250b4ec, State: Initialized, Role: FOLLOWER
I20260924 15:23:17.230458 7391 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425c5da2ae4ea59b92ec828250b4ec" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43391 } }
I20260924 15:23:17.230538 7391 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:17.230561 7391 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:17.230633 7391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:17.231326 7391 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425c5da2ae4ea59b92ec828250b4ec" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43391 } }
I20260924 15:23:17.231441 7391 leader_election.cc:302] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28425c5da2ae4ea59b92ec828250b4ec; no voters:
I20260924 15:23:17.231623 7391 leader_election.cc:288] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:17.231683 7396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:17.231837 7396 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [term 1 LEADER]: Becoming Leader. State: Replica: 28425c5da2ae4ea59b92ec828250b4ec, State: Running, Role: LEADER
I20260924 15:23:17.231889 7391 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:17.231959 7396 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425c5da2ae4ea59b92ec828250b4ec" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43391 } }
I20260924 15:23:17.232319 7396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28425c5da2ae4ea59b92ec828250b4ec. Latest consensus state: current_term: 1 leader_uuid: "28425c5da2ae4ea59b92ec828250b4ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425c5da2ae4ea59b92ec828250b4ec" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43391 } } }
I20260924 15:23:17.232551 7396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:17.232475 7397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28425c5da2ae4ea59b92ec828250b4ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28425c5da2ae4ea59b92ec828250b4ec" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43391 } } }
I20260924 15:23:17.232949 7397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:17.233064 7405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:17.233592 7405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:17.233891 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:17.222220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49520 (local address 127.4.78.126:43391)
0924 15:23:17.222421 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:17.222431 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:17.222472 (+ 41us) server_negotiation.cc:407] Connection header received
0924 15:23:17.223051 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:17.223059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:17.223128 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:17.223272 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:17.224058 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:17.225500 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:17.226597 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:17.227220 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:17.227329 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:17.229596 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:17.229601 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:17.229601 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:17.229616 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:17.231713 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:17.232160 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:17.232162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:17.232163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:17.232264 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:17.232497 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:17.232498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:17.232498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:17.232832 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:17.233418 (+ 586us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:17.233500 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:23:17.233641 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260924 15:23:17.235025 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:23:17.235280 7405 catalog_manager.cc:1347] Generated new cluster ID: f949f847a18c47439c253c8cc5eded54
I20260924 15:23:17.235318 7405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:17.246779 7405 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:17.247287 7405 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:17.257283 7394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:17.253157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49534 (local address 127.4.78.126:43391)
0924 15:23:17.253193 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:17.253195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:17.253343 (+ 148us) server_negotiation.cc:407] Connection header received
0924 15:23:17.253398 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:17.253399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:17.253410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:17.253472 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:17.253892 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:17.254685 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:17.255719 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:17.256044 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:17.256109 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:17.256813 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:17.256814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:17.256814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:17.257022 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:17.257041 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:17.257048 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:17.257111 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:17.257175 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 15:23:17.257201 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:23:17.257645 7405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28425c5da2ae4ea59b92ec828250b4ec: Generated new TSK 0
I20260924 15:23:17.257912 7405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:23:17.260975 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 7322
2026-09-24T15:23:17Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1292 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:17 dist-test-slave-l766 krb5kdc[7424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:17 dist-test-slave-l766 krb5kdc[7424](info): set up 2 sockets
Sep 24 15:23:17 dist-test-slave-l766 krb5kdc[7424](info): commencing operation
krb5kdc: starting...
W20260924 15:23:18.289937 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:18 dist-test-slave-l766 krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:18.409566 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42963
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:42963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:18.489073 7437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:18.489279 7437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:18.489300 7437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:18.490406 7437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:18.490438 7437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:18.490453 7437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:18.490465 7437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:18.490477 7437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:18.491892 7437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:42963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:18.492084 7437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:18.492242 7437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:18.497009 7443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:18.497056 7442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:18.497026 7437 server_base.cc:1034] running on GCE node
W20260924 15:23:18.497020 7445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:18.497479 7437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:18.497768 7437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:18.498951 7437 hybrid_clock.cc:648] HybridClock initialized: now 1790263398498932 us; error 36 us; skew 500 ppm
Sep 24 15:23:18 dist-test-slave-l766 krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:18.501395 7437 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:18.501888 7437 webserver.cc:460] Webserver started at http://127.4.78.126:45801/ using document root <none> and password file <none>
I20260924 15:23:18.502105 7437 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:18.502167 7437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:18.502311 7437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:18.503326 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e5f774b688234258a92f83730163ac34"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "2174a08f33a2ddfdfd4fb3ca531c371a"
server_key_iv: "a17cbbea54bf44d899abdc9c2b48a4a5"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.503693 7437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e5f774b688234258a92f83730163ac34"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "2174a08f33a2ddfdfd4fb3ca531c371a"
server_key_iv: "a17cbbea54bf44d899abdc9c2b48a4a5"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.504983 7437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:18.505652 7452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:18.505859 7437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:18.505961 7437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e5f774b688234258a92f83730163ac34"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "2174a08f33a2ddfdfd4fb3ca531c371a"
server_key_iv: "a17cbbea54bf44d899abdc9c2b48a4a5"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.506043 7437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:18.529026 7437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:18.531464 7437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:18.531623 7437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:18.535514 7437 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:42963
I20260924 15:23:18.535552 7504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:42963 every 8 connection(s)
I20260924 15:23:18.535913 7437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:18.536897 7505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:18.539145 7505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34: Bootstrap starting.
I20260924 15:23:18.539666 7505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:18.539898 7505 log.cc:824] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:18.540518 7505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34: No bootstrap required, opened a new log
I20260924 15:23:18.542083 7505 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f774b688234258a92f83730163ac34" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42963 } }
I20260924 15:23:18.542214 7505 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:18.542236 7505 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f774b688234258a92f83730163ac34, State: Initialized, Role: FOLLOWER
I20260924 15:23:18.542340 7505 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f774b688234258a92f83730163ac34" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42963 } }
I20260924 15:23:18.542387 7505 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:18.542407 7505 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:18.542433 7505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:18.543141 7505 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f774b688234258a92f83730163ac34" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42963 } }
I20260924 15:23:18.543226 7505 leader_election.cc:302] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f774b688234258a92f83730163ac34; no voters:
I20260924 15:23:18.543335 7505 leader_election.cc:288] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:18.543390 7508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:18.543547 7508 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [term 1 LEADER]: Becoming Leader. State: Replica: e5f774b688234258a92f83730163ac34, State: Running, Role: LEADER
I20260924 15:23:18.543603 7505 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:18.543689 7508 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f774b688234258a92f83730163ac34" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42963 } }
I20260924 15:23:18.544183 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 7437
I20260924 15:23:18.544314 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:18.544519 4409 external_mini_cluster.cc:1428] Setting key 0b5e8aa51988f7d7d76599e079361d30
I20260924 15:23:18.544947 7510 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f774b688234258a92f83730163ac34. Latest consensus state: current_term: 1 leader_uuid: "e5f774b688234258a92f83730163ac34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f774b688234258a92f83730163ac34" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42963 } } }
I20260924 15:23:18.545184 7510 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:18.545189 7509 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f774b688234258a92f83730163ac34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f774b688234258a92f83730163ac34" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42963 } } }
I20260924 15:23:18.545401 7509 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34 [sys.catalog]: This master's current role is: LEADER
W20260924 15:23:18.545681 7523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:23:18.545796 7523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:23:18.545889 7526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:18.546052 7526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:18.547505 7526 catalog_manager.cc:1347] Generated new cluster ID: c9f0acb6c74e4ee4a96045a6aa8dea1f
I20260924 15:23:18.547554 7526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:23:18 dist-test-slave-l766 krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:18.555965 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:18.545119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52994 (local address 127.4.78.126:42963)
0924 15:23:18.545282 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:18.545291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:18.545324 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:23:18.545435 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:18.545442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:18.545500 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:18.545673 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:18.546506 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.547728 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.548809 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.549423 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.549524 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:18.552279 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:18.552284 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:18.552284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:18.552297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:18.554313 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:18.554698 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:18.554701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:18.554701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:18.554831 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:18.555021 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:18.555022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:18.555022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:18.555347 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:18.555502 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:18.555598 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:23:18.555732 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260924 15:23:18.573966 7526 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:18.574474 7526 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:18.585565 7526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f774b688234258a92f83730163ac34: Generated new TSK 0
I20260924 15:23:18.585778 7526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:18.613341 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42963
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:18.692500 7531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:18.692718 7531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:18.692757 7531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:18.693917 7531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:18.693977 7531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:18.694022 7531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:18.695380 7531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:18.695621 7531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:18.695827 7531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:18.700567 7537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:18.700569 7539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:18.700570 7536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:18.700837 7531 server_base.cc:1034] running on GCE node
I20260924 15:23:18.701009 7531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:18.701191 7531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:18.702320 7531 hybrid_clock.cc:648] HybridClock initialized: now 1790263398702296 us; error 39 us; skew 500 ppm
Sep 24 15:23:18 dist-test-slave-l766 krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:18.704670 7531 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:18.705104 7531 webserver.cc:460] Webserver started at http://127.4.78.65:42607/ using document root <none> and password file <none>
I20260924 15:23:18.705353 7531 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:18.705400 7531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:18.705554 7531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:18.706640 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "da5f9d33d7d64806ae16ac94ebc910c0"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "e42a48bc5718fb0b0c2bd7de91355019"
server_key_iv: "e4ad86b180a9930fda92cd840c7a7d74"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.706979 7531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "da5f9d33d7d64806ae16ac94ebc910c0"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "e42a48bc5718fb0b0c2bd7de91355019"
server_key_iv: "e4ad86b180a9930fda92cd840c7a7d74"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.708221 7531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:23:18.708954 7546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:18.709121 7531 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:18.709223 7531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "da5f9d33d7d64806ae16ac94ebc910c0"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "e42a48bc5718fb0b0c2bd7de91355019"
server_key_iv: "e4ad86b180a9930fda92cd840c7a7d74"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.709304 7531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:18.719623 7531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:18.721359 7531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:18.721513 7531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:18.721719 7531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:18.722028 7531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:18.722078 7531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:18.722118 7531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:18.722144 7531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:18.727727 7531 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:35009
I20260924 15:23:18.727771 7659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:35009 every 8 connection(s)
I20260924 15:23:18.728060 7531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:23:18 dist-test-slave-l766 krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:18.736486 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:18.728712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:41541 (local address 127.4.78.126:42963)
0924 15:23:18.728778 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:18.728781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:18.728970 (+ 189us) server_negotiation.cc:407] Connection header received
0924 15:23:18.729450 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:18.729451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:18.729464 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:18.729531 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:18.730563 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.731060 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.731994 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.732193 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.732242 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:18.734204 (+ 1962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:18.734206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:18.734206 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:18.734214 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:18.735417 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:18.735762 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:18.735763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:18.735763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:18.735812 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:18.736009 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:18.736010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:18.736010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:18.736105 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:18.736153 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:18.736361 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:23:18.736425 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:23:18.736691 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:18.728689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42963 (local address 127.4.78.65:41541)
0924 15:23:18.728871 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:18.728904 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:18.729370 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:18.729622 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:18.729627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:18.730293 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:18.730482 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:18.730486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.731135 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.731135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:18.731924 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:18.731926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.732261 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.732263 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:18.732357 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:18.732809 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:18.732817 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:18.734121 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:18.735487 (+ 1366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:18.735493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:18.735494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:18.735701 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:18.735866 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:18.735867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:18.735867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:18.735959 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:18.736154 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:18.736158 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:18.736261 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 15:23:18.736338 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:23:18.736451 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260924 15:23:18.737193 7660 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42963
I20260924 15:23:18.737294 7660 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:18.737463 7660 heartbeater.cc:511] Master 127.4.78.126:42963 requested a full tablet report, sending...
I20260924 15:23:18.738029 7469 ts_manager.cc:194] Registered new tserver with Master: da5f9d33d7d64806ae16ac94ebc910c0 (127.4.78.65:35009)
I20260924 15:23:18.738089 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 7531
I20260924 15:23:18.738193 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:18.738402 4409 external_mini_cluster.cc:1428] Setting key ce0062967d32d1212601fdf4bb1f7a33
I20260924 15:23:18.738893 7469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:41541
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:18.760692 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42963
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:18.839766 7665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:18.839980 7665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:18.840018 7665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:18.841217 7665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:18.841279 7665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:18.841322 7665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:18.842751 7665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:18.842993 7665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:18.843199 7665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:23:18.848014 7665 server_base.cc:1034] running on GCE node
W20260924 15:23:18.847972 7670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:18.848049 7673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:18.847987 7671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:18.848443 7665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:18.848685 7665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:18.849862 7665 hybrid_clock.cc:648] HybridClock initialized: now 1790263398849846 us; error 27 us; skew 500 ppm
Sep 24 15:23:18 dist-test-slave-l766 krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:18.852192 7665 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:18.852595 7665 webserver.cc:460] Webserver started at http://127.4.78.66:41683/ using document root <none> and password file <none>
I20260924 15:23:18.852772 7665 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:18.852810 7665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:18.852893 7665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:18.853909 7665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "10a320e72c0f4ff8a11f4bd77ee4dbe2"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "d3bef260b52c1570c40ff93c23a750c4"
server_key_iv: "1d804ac5260dc0239fd2b5785c247b92"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.854208 7665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "10a320e72c0f4ff8a11f4bd77ee4dbe2"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "d3bef260b52c1570c40ff93c23a750c4"
server_key_iv: "1d804ac5260dc0239fd2b5785c247b92"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.855451 7665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:18.856099 7680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:18.856267 7665 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:18.856350 7665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "10a320e72c0f4ff8a11f4bd77ee4dbe2"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "d3bef260b52c1570c40ff93c23a750c4"
server_key_iv: "1d804ac5260dc0239fd2b5785c247b92"
server_key_version: "encryptionkey@0"
I20260924 15:23:18.856417 7665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:18.868666 7665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:18.870486 7665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:18.870635 7665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:18.870855 7665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:18.871146 7665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:18.871181 7665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:18.871203 7665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:18.871248 7665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:18.876940 7665 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:40711
I20260924 15:23:18.876991 7793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:40711 every 8 connection(s)
I20260924 15:23:18.877290 7665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:23:18 dist-test-slave-l766 krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:18.884711 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 7665
I20260924 15:23:18.884801 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:18.885008 4409 external_mini_cluster.cc:1428] Setting key f994d84a9f063f5aee25d316098d7aee
I20260924 15:23:18.886015 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:18.877977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:39401 (local address 127.4.78.126:42963)
0924 15:23:18.878022 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:18.878024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:18.878222 (+ 198us) server_negotiation.cc:407] Connection header received
0924 15:23:18.878712 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:18.878713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:18.878724 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:18.878776 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:18.879652 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.880141 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.881072 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.881253 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.881323 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:18.883316 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:18.883318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:18.883318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:18.883324 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:18.884492 (+ 1168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:18.884952 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:18.884954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:18.884954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:18.885018 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:18.885309 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:18.885310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:18.885311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:18.885451 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:18.885509 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:18.885839 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 15:23:18.885927 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:23:18.886193 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:18.877963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42963 (local address 127.4.78.66:39401)
0924 15:23:18.878116 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:18.878152 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:18.878626 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:18.878827 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:18.878830 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:18.879387 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:18.879578 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:18.879582 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.880217 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.880218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:18.880997 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:18.880999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:18.881338 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:18.881338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:18.881434 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:18.881947 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:18.881955 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:18.883234 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:18.884560 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:18.884565 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:18.884567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:18.884817 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:18.885085 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:18.885086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:18.885086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:18.885222 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:18.885516 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:18.885522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:18.885683 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 15:23:18.885796 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 15:23:18.885969 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 15:23:18.886581 7794 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42963
I20260924 15:23:18.886685 7794 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:18.886879 7794 heartbeater.cc:511] Master 127.4.78.126:42963 requested a full tablet report, sending...
I20260924 15:23:18.887244 7469 ts_manager.cc:194] Registered new tserver with Master: 10a320e72c0f4ff8a11f4bd77ee4dbe2 (127.4.78.66:40711)
I20260924 15:23:18.887768 7469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:39401
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:18.909866 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42963
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:18.990284 7799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:18.990491 7799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:18.990514 7799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:18.991693 7799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:18.991737 7799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:18.991771 7799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:18.993281 7799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:18.993525 7799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:18.993742 7799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:18.998634 7807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:18.998615 7804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:18.998718 7805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:18.998776 7799 server_base.cc:1034] running on GCE node
I20260924 15:23:18.999099 7799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:18.999359 7799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:19.000509 7799 hybrid_clock.cc:648] HybridClock initialized: now 1790263399000495 us; error 35 us; skew 500 ppm
Sep 24 15:23:18 dist-test-slave-l766 krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:19.003084 7799 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:19.003535 7799 webserver.cc:460] Webserver started at http://127.4.78.67:39007/ using document root <none> and password file <none>
I20260924 15:23:19.003769 7799 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:19.003834 7799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:19.003975 7799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:19.004983 7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "98f1bec8a40448f5a63bcbd2567ee12e"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "5219d6ebcffb918b2d229f2a45d54644"
server_key_iv: "32e04777f9f2d7b5ec4a6b75b9f2490a"
server_key_version: "encryptionkey@0"
I20260924 15:23:19.005367 7799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "98f1bec8a40448f5a63bcbd2567ee12e"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "5219d6ebcffb918b2d229f2a45d54644"
server_key_iv: "32e04777f9f2d7b5ec4a6b75b9f2490a"
server_key_version: "encryptionkey@0"
I20260924 15:23:19.006873 7799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:19.007709 7814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:19.007908 7799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:19.008030 7799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "98f1bec8a40448f5a63bcbd2567ee12e"
format_stamp: "Formatted at 2026-09-24 15:23:18 on dist-test-slave-l766"
server_key: "5219d6ebcffb918b2d229f2a45d54644"
server_key_iv: "32e04777f9f2d7b5ec4a6b75b9f2490a"
server_key_version: "encryptionkey@0"
I20260924 15:23:19.008114 7799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:19.019855 7799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:19.021744 7799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:19.021972 7799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:19.022161 7799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:19.022449 7799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:19.022481 7799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:19.022521 7799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:19.022555 7799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:19.028304 7799 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:43013
I20260924 15:23:19.028354 7927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:43013 every 8 connection(s)
I20260924 15:23:19.028647 7799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:19.034384 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 7799
I20260924 15:23:19.034488 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:19.034711 4409 external_mini_cluster.cc:1428] Setting key 7833fcc1e5d1bba10708b5006fff6c6e
Sep 24 15:23:19 dist-test-slave-l766 krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263398, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:19.039017 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:19.029361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:53579 (local address 127.4.78.126:42963)
0924 15:23:19.029420 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:19.029423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:19.029675 (+ 252us) server_negotiation.cc:407] Connection header received
0924 15:23:19.030182 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:19.030183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:19.030198 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:19.030269 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:19.031620 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:19.032318 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:19.033951 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:19.034147 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:19.034207 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:19.036561 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:19.036563 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:19.036563 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:19.036570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:19.037786 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:19.038231 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:19.038232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:19.038233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:19.038287 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:19.038495 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:19.038496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:19.038496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:19.038614 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:19.038668 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:19.038889 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 15:23:19.038954 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:23:19.039162 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:19.029344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42963 (local address 127.4.78.67:53579)
0924 15:23:19.029520 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:19.029560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:19.030060 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:19.030373 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:19.030391 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:19.031228 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:19.031509 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:19.031514 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:19.032435 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:19.032436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:19.033855 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:19.033858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:19.034237 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:19.034238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:19.034403 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:19.034959 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:19.034968 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:19.036476 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:19.037939 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:19.037945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:19.037947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:19.038149 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:19.038342 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:19.038343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:19.038343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:19.038440 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:19.038668 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:19.038673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:19.038779 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 15:23:19.038859 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:23:19.038971 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":100,"threads_started":1}
I20260924 15:23:19.039608 7928 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42963
I20260924 15:23:19.039729 7928 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:19.039911 7928 heartbeater.cc:511] Master 127.4.78.126:42963 requested a full tablet report, sending...
I20260924 15:23:19.040297 7469 ts_manager.cc:194] Registered new tserver with Master: 98f1bec8a40448f5a63bcbd2567ee12e (127.4.78.67:43013)
I20260924 15:23:19.040683 7469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:53579
I20260924 15:23:19.046531 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:23:19 dist-test-slave-l766 krb5kdc[7424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263399, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 15:23:19 dist-test-slave-l766 krb5kdc[7424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263399, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:19.068271 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:19.061384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.4.78.126:42963)
0924 15:23:19.061484 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:19.061487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:19.061523 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:23:19.061579 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:19.061580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:19.061589 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:19.061652 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:19.062231 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:19.062737 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:19.063420 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:19.063607 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:19.063672 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:19.065880 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:19.065882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:19.065882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:19.065889 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:19.067297 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:19.067586 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:19.067587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:19.067587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:19.067645 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:19.067791 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:19.067791 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:19.067792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:19.067923 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:19.067991 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:19.068071 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:23:19.068152 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 15:23:19.068500 7469 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52998
I20260924 15:23:19.068868 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 7531
I20260924 15:23:19.073269 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 7665
I20260924 15:23:19.077706 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 7799
I20260924 15:23:19.081964 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 7437
2026-09-24T15:23:19Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1823 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:19 dist-test-slave-l766 krb5kdc[7940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:19 dist-test-slave-l766 krb5kdc[7940](info): set up 2 sockets
Sep 24 15:23:19 dist-test-slave-l766 krb5kdc[7940](info): commencing operation
krb5kdc: starting...
W20260924 15:23:20.116643 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:20.232404 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42749
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:42749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:20.313208 7953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:20.313409 7953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:20.313431 7953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:20.314595 7953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:20.314636 7953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:20.314651 7953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:20.314662 7953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:20.314713 7953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:20.316191 7953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:42749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:20.316453 7953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:20.316687 7953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:20.321300 7961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:20.321453 7953 server_base.cc:1034] running on GCE node
W20260924 15:23:20.321319 7959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:20.321310 7958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:20.321903 7953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:20.322196 7953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:20.323354 7953 hybrid_clock.cc:648] HybridClock initialized: now 1790263400323354 us; error 40 us; skew 500 ppm
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:20.330036 7953 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:20.330452 7953 webserver.cc:460] Webserver started at http://127.4.78.126:45329/ using document root <none> and password file <none>
I20260924 15:23:20.330670 7953 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:20.330734 7953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:20.330873 7953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:20.331879 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "540a6f22c3c8478d8478218cd78dceb7"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "4afcf427f0ff0d6833927b86ce229e06"
server_key_iv: "258bd17414bc6ef78e8603c4cbaaa057"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.332242 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "540a6f22c3c8478d8478218cd78dceb7"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "4afcf427f0ff0d6833927b86ce229e06"
server_key_iv: "258bd17414bc6ef78e8603c4cbaaa057"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.333599 7953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:20.334347 7968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.334510 7953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.334602 7953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "540a6f22c3c8478d8478218cd78dceb7"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "4afcf427f0ff0d6833927b86ce229e06"
server_key_iv: "258bd17414bc6ef78e8603c4cbaaa057"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.334681 7953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:20.352058 7953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:20.354233 7953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:20.354384 7953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:20.358218 7953 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:42749
I20260924 15:23:20.358251 8020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:42749 every 8 connection(s)
I20260924 15:23:20.358708 7953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:20.359326 8021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:20.361580 8021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7: Bootstrap starting.
I20260924 15:23:20.362241 8021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:20.362569 8021 log.cc:824] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:20.363266 8021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7: No bootstrap required, opened a new log
I20260924 15:23:20.364800 8021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540a6f22c3c8478d8478218cd78dceb7" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42749 } }
I20260924 15:23:20.364933 8021 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:20.364987 8021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540a6f22c3c8478d8478218cd78dceb7, State: Initialized, Role: FOLLOWER
I20260924 15:23:20.365125 8021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540a6f22c3c8478d8478218cd78dceb7" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42749 } }
I20260924 15:23:20.365206 8021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:20.365265 8021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:20.365312 8021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:20.366252 8021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540a6f22c3c8478d8478218cd78dceb7" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42749 } }
I20260924 15:23:20.366369 8021 leader_election.cc:302] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 540a6f22c3c8478d8478218cd78dceb7; no voters:
I20260924 15:23:20.366544 8021 leader_election.cc:288] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:20.366585 8024 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:20.366761 8024 raft_consensus.cc:686] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [term 1 LEADER]: Becoming Leader. State: Replica: 540a6f22c3c8478d8478218cd78dceb7, State: Running, Role: LEADER
I20260924 15:23:20.366796 8021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:20.366889 8024 consensus_queue.cc:237] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540a6f22c3c8478d8478218cd78dceb7" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42749 } }
I20260924 15:23:20.367064 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 7953
I20260924 15:23:20.367187 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:20.367419 4409 external_mini_cluster.cc:1428] Setting key 60d6de0ddad5274219b851ace408b42c
I20260924 15:23:20.367919 8025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "540a6f22c3c8478d8478218cd78dceb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540a6f22c3c8478d8478218cd78dceb7" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42749 } } }
I20260924 15:23:20.368012 8025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:20.368374 8026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 540a6f22c3c8478d8478218cd78dceb7. Latest consensus state: current_term: 1 leader_uuid: "540a6f22c3c8478d8478218cd78dceb7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "540a6f22c3c8478d8478218cd78dceb7" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42749 } } }
I20260924 15:23:20.368594 8026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7 [sys.catalog]: This master's current role is: LEADER
W20260924 15:23:20.368703 8042 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:23:20.368767 8042 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:23:20.368860 8040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:20.369033 8040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:20.370393 8040 catalog_manager.cc:1347] Generated new cluster ID: 12a8c247f4e846c4b21e62200c463d03
I20260924 15:23:20.370441 8040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:20.379360 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.368031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34052 (local address 127.4.78.126:42749)
0924 15:23:20.368281 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:20.368290 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:20.368645 (+ 355us) server_negotiation.cc:407] Connection header received
0924 15:23:20.368765 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:20.368772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:20.368827 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:20.368952 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:20.369891 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.371283 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.372302 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.372969 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.373083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.375824 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:20.375828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:20.375828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:20.375839 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:20.377689 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.378125 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.378128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.378128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.378240 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.378457 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.378457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.378458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.378751 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:20.378887 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:20.379020 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 15:23:20.379148 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260924 15:23:20.384946 8040 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:20.385442 8040 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:20.393100 8040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 540a6f22c3c8478d8478218cd78dceb7: Generated new TSK 0
I20260924 15:23:20.393278 8040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:20.426501 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42749
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:20.506847 8047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:20.507071 8047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:20.507108 8047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:20.508366 8047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:20.508433 8047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:20.508479 8047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:20.510000 8047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:20.510257 8047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:20.510480 8047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:23:20.515429 8047 server_base.cc:1034] running on GCE node
W20260924 15:23:20.515393 8052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:20.515393 8055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:20.515398 8053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:20.515904 8047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:20.516150 8047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:20.517310 8047 hybrid_clock.cc:648] HybridClock initialized: now 1790263400517300 us; error 40 us; skew 500 ppm
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:20.519723 8047 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:20.520226 8047 webserver.cc:460] Webserver started at http://127.4.78.65:34555/ using document root <none> and password file <none>
I20260924 15:23:20.520447 8047 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:20.520510 8047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:20.520640 8047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:20.521616 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "5e5635bcf8254b408e73de431bd5ef84"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "ece57bef593bd501ae7935c2251b0c54"
server_key_iv: "c130757655f1bd1d6b906a1a1c07ae97"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.521986 8047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "5e5635bcf8254b408e73de431bd5ef84"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "ece57bef593bd501ae7935c2251b0c54"
server_key_iv: "c130757655f1bd1d6b906a1a1c07ae97"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.523267 8047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:20.523962 8062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.524132 8047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:20.524235 8047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "5e5635bcf8254b408e73de431bd5ef84"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "ece57bef593bd501ae7935c2251b0c54"
server_key_iv: "c130757655f1bd1d6b906a1a1c07ae97"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.524315 8047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:20.555968 8047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:20.557760 8047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:20.557930 8047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:20.558173 8047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:20.558472 8047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:20.558523 8047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.558564 8047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:20.558593 8047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.564365 8047 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:35265
I20260924 15:23:20.564414 8175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:35265 every 8 connection(s)
I20260924 15:23:20.564740 8047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:20.571769 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 8047
I20260924 15:23:20.571878 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:20.572096 4409 external_mini_cluster.cc:1428] Setting key c6cf51c57311ff2b84531fe80f31267e
I20260924 15:23:20.574368 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.565437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:38585 (local address 127.4.78.126:42749)
0924 15:23:20.565497 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:20.565499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:20.565717 (+ 218us) server_negotiation.cc:407] Connection header received
0924 15:23:20.566745 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:20.566746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:20.566763 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:20.566814 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:20.567443 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.568014 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.569050 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.569277 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.569347 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.571370 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:20.571372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:20.571372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:20.571381 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:20.572932 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.573310 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.573311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.573312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.573372 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.573616 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.573617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.573617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.573739 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:20.573815 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:20.574220 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 15:23:20.574304 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:23:20.574626 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.565411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42749 (local address 127.4.78.65:38585)
0924 15:23:20.565593 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:20.565629 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:20.566590 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:20.566888 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:20.566892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:20.567146 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:20.567363 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:20.567370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.568101 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.568102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:20.568949 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:20.568951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.569353 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.569354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:20.569473 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.569955 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:20.569963 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:20.571275 (+ 1312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:20.573005 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:20.573012 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:20.573014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:20.573237 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:20.573435 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:20.573436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:20.573437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:20.573544 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:20.573863 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:20.573871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:20.574046 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 15:23:20.574195 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 15:23:20.574372 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260924 15:23:20.575134 8176 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42749
I20260924 15:23:20.575227 8176 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:20.575413 8176 heartbeater.cc:511] Master 127.4.78.126:42749 requested a full tablet report, sending...
I20260924 15:23:20.575892 7985 ts_manager.cc:194] Registered new tserver with Master: 5e5635bcf8254b408e73de431bd5ef84 (127.4.78.65:35265)
I20260924 15:23:20.576521 7985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:38585
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:20.594902 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42749
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:20.673888 8181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:20.674079 8181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:20.674100 8181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:20.675292 8181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:20.675356 8181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:20.675381 8181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:20.676841 8181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:20.677057 8181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:20.677268 8181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:20.682052 8189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:20.682063 8186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:20.682068 8187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:20.682322 8181 server_base.cc:1034] running on GCE node
I20260924 15:23:20.682498 8181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:20.682715 8181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:20.683866 8181 hybrid_clock.cc:648] HybridClock initialized: now 1790263400683857 us; error 36 us; skew 500 ppm
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:20.686204 8181 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:20.686558 8181 webserver.cc:460] Webserver started at http://127.4.78.66:36685/ using document root <none> and password file <none>
I20260924 15:23:20.686728 8181 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:20.686775 8181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:20.686864 8181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:20.687839 8181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "5397c48cb65c4feaa227068f2f15555d"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "bc9c32f6042bae9e83d7d5ee2b0e1b91"
server_key_iv: "2893de74eafbcef0c020da377309bbd1"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.688138 8181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "5397c48cb65c4feaa227068f2f15555d"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "bc9c32f6042bae9e83d7d5ee2b0e1b91"
server_key_iv: "2893de74eafbcef0c020da377309bbd1"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.689350 8181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:23:20.689990 8196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.690155 8181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.690243 8181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "5397c48cb65c4feaa227068f2f15555d"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "bc9c32f6042bae9e83d7d5ee2b0e1b91"
server_key_iv: "2893de74eafbcef0c020da377309bbd1"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.690323 8181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:20.702214 8181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:20.704452 8181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:20.704629 8181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:20.704859 8181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:20.705165 8181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:20.705216 8181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.705256 8181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:20.705284 8181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.711053 8181 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:41285
I20260924 15:23:20.711097 8309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:41285 every 8 connection(s)
I20260924 15:23:20.711392 8181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:20.719199 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 8181
I20260924 15:23:20.719311 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:20.719529 4409 external_mini_cluster.cc:1428] Setting key 96b618dc2e0184b4a9fdffc4012431bb
I20260924 15:23:20.720472 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.712041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:42477 (local address 127.4.78.126:42749)
0924 15:23:20.712101 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:20.712103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:20.712297 (+ 194us) server_negotiation.cc:407] Connection header received
0924 15:23:20.713236 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:20.713237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:20.713247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:20.713311 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:20.713936 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.714447 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.715392 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.715576 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.715627 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.717519 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:20.717521 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:20.717521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:20.717528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:20.718918 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.719417 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.719418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.719419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.719469 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.719740 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.719741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.719741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.719889 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:20.719955 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:20.720219 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:23:20.720318 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:23:20.720613 8312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.712020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42749 (local address 127.4.78.66:42477)
0924 15:23:20.712169 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:20.712211 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:20.713134 (+ 923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:20.713379 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:20.713383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:20.713641 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:20.713859 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:20.713863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.714530 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.714531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:20.715317 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:20.715318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.715642 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.715643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:20.715745 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.716159 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:20.716167 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:20.717437 (+ 1270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:20.718988 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:20.718994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:20.718996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:20.719337 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:20.719533 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:20.719534 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:20.719534 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:20.719669 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:20.719945 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:20.719950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:20.720060 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 15:23:20.720195 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 15:23:20.720364 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260924 15:23:20.721047 8310 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42749
I20260924 15:23:20.721151 8310 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:20.721321 8310 heartbeater.cc:511] Master 127.4.78.126:42749 requested a full tablet report, sending...
I20260924 15:23:20.721751 7985 ts_manager.cc:194] Registered new tserver with Master: 5397c48cb65c4feaa227068f2f15555d (127.4.78.66:41285)
I20260924 15:23:20.722182 7985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:42477
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:20.742501 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42749
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:20.820921 8315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:20.821099 8315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:20.821122 8315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:20.822257 8315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:20.822296 8315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:20.822345 8315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:20.823868 8315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:20.824095 8315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:20.824337 8315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:20.828970 8320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:20.828992 8321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:20.828966 8323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:20.829311 8315 server_base.cc:1034] running on GCE node
I20260924 15:23:20.829520 8315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:20.829734 8315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:20.830884 8315 hybrid_clock.cc:648] HybridClock initialized: now 1790263400830870 us; error 29 us; skew 500 ppm
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:20.833222 8315 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:20.833663 8315 webserver.cc:460] Webserver started at http://127.4.78.67:39653/ using document root <none> and password file <none>
I20260924 15:23:20.833947 8315 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:20.833998 8315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:20.834134 8315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:20.835103 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "0cdafcb48bc04d9e834cd261e38b88e5"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "08abb5c50eed4300b2436705f2fc590a"
server_key_iv: "5fc7a21f34f1b6952bbe7741870b045b"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.835449 8315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "0cdafcb48bc04d9e834cd261e38b88e5"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "08abb5c50eed4300b2436705f2fc590a"
server_key_iv: "5fc7a21f34f1b6952bbe7741870b045b"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.836683 8315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:20.837363 8330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.837549 8315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:20.837637 8315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "0cdafcb48bc04d9e834cd261e38b88e5"
format_stamp: "Formatted at 2026-09-24 15:23:20 on dist-test-slave-l766"
server_key: "08abb5c50eed4300b2436705f2fc590a"
server_key_iv: "5fc7a21f34f1b6952bbe7741870b045b"
server_key_version: "encryptionkey@0"
I20260924 15:23:20.837723 8315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:20.863991 8315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:20.865671 8315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:20.865860 8315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:20.866067 8315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:20.866427 8315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:20.866470 8315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.866514 8315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:20.866542 8315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:20.872392 8315 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:41715
I20260924 15:23:20.872450 8443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:41715 every 8 connection(s)
I20260924 15:23:20.872741 8315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:20.876734 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 8315
I20260924 15:23:20.876852 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:20.877053 4409 external_mini_cluster.cc:1428] Setting key 22819fef24c7692a98694d2fd8d67320
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:20.881675 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.873381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:46029 (local address 127.4.78.126:42749)
0924 15:23:20.873421 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:20.873423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:20.873616 (+ 193us) server_negotiation.cc:407] Connection header received
0924 15:23:20.874606 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:20.874606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:20.874615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:20.874681 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:20.875228 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.875692 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.876670 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.876874 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.876935 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.879548 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:20.879549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:20.879549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:20.879556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:20.880637 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.880992 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.880993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.880993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.881040 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.881227 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.881228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.881228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.881317 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:20.881363 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:20.881557 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 15:23:20.881609 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:23:20.881906 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.873349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42749 (local address 127.4.78.67:46029)
0924 15:23:20.873497 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:20.873538 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:20.874519 (+ 981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:20.874737 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:20.874740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:20.874977 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:20.875156 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:20.875160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.875768 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.875768 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:20.876587 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:20.876589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.876945 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.876946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:20.877073 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.877615 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:20.877625 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:20.879481 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:20.880710 (+ 1229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:20.880715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:20.880717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:20.880932 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:20.881090 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:20.881091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:20.881091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:20.881181 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:20.881365 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:20.881369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:20.881470 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 15:23:20.881545 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:23:20.881651 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260924 15:23:20.882318 8444 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42749
I20260924 15:23:20.882393 8444 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:20.882567 8444 heartbeater.cc:511] Master 127.4.78.126:42749 requested a full tablet report, sending...
I20260924 15:23:20.882886 7985 ts_manager.cc:194] Registered new tserver with Master: 0cdafcb48bc04d9e834cd261e38b88e5 (127.4.78.67:41715)
I20260924 15:23:20.883200 7985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:46029
I20260924 15:23:20.888677 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:20.910430 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.903088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53804 (local address 127.4.78.65:35265)
0924 15:23:20.903279 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:20.903284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:20.903294 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:20.903320 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:20.903322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:20.903337 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:20.903402 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:20.903918 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.904625 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.905320 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.905627 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.905680 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.907785 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:20.907794 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:20.907795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:20.907801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:20.909374 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.909662 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.909665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.909665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.909718 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.909906 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.909907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.909907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.910035 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:20.910144 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:20.910198 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 15:23:20.910275 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263400, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:20.917517 8039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:20.911637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.4.78.126:42749)
0924 15:23:20.911685 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:20.911687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:20.911694 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:20.911753 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:20.911754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:20.911761 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:20.911808 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:20.912231 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.912695 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.913310 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:20.913480 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:20.913528 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:20.915559 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:20.915561 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:20.915561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:20.915567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:20.916670 (+ 1103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.916951 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.916952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.916952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.916997 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:20.917135 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:20.917136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:20.917136 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:20.917257 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:20.917322 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:20.917404 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:23:20.917458 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 15:23:20.917765 7985 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34068
I20260924 15:23:20.918121 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 8047
I20260924 15:23:20.922981 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 8181
I20260924 15:23:20.928834 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 8315
I20260924 15:23:20.933146 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 7953
2026-09-24T15:23:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1850 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[8456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[8456](info): set up 2 sockets
Sep 24 15:23:20 dist-test-slave-l766 krb5kdc[8456](info): commencing operation
krb5kdc: starting...
W20260924 15:23:21.964643 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:22.079943 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:40677
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:40677
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:22.158195 8469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:22.158427 8469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:22.158463 8469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:22.159636 8469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:22.159701 8469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:22.159724 8469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:22.159744 8469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:22.159775 8469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:22.161193 8469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:40677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:40677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:22.161454 8469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:22.161660 8469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:22.166205 8475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:22.166206 8477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:22.166225 8474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:22.166689 8469 server_base.cc:1034] running on GCE node
I20260924 15:23:22.166968 8469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:22.167303 8469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:22.168457 8469 hybrid_clock.cc:648] HybridClock initialized: now 1790263402168444 us; error 25 us; skew 500 ppm
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:22.170761 8469 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:22.171105 8469 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:22.171170 8469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:22.171310 8469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:22.172343 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "c605a64c7f574743ace48b98d71248c1"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "3a0de5a6771064740d04f43cf8b597e1"
server_key_iv: "41fd8ce615c5da7d52676f352e91c437"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.172700 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "c605a64c7f574743ace48b98d71248c1"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "3a0de5a6771064740d04f43cf8b597e1"
server_key_iv: "41fd8ce615c5da7d52676f352e91c437"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.173990 8469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:22.174674 8483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.174829 8469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:22.174927 8469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "c605a64c7f574743ace48b98d71248c1"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "3a0de5a6771064740d04f43cf8b597e1"
server_key_iv: "41fd8ce615c5da7d52676f352e91c437"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.175000 8469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:22.183014 8469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:22.184787 8469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:22.184935 8469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:22.188719 8469 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:40677
I20260924 15:23:22.188746 8535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:40677 every 8 connection(s)
I20260924 15:23:22.189030 8469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:22.189966 8536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:22.192152 8536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1: Bootstrap starting.
I20260924 15:23:22.192677 8536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:22.192917 8536 log.cc:824] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:22.193512 8536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1: No bootstrap required, opened a new log
I20260924 15:23:22.194196 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 8469
I20260924 15:23:22.194308 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:22.194489 4409 external_mini_cluster.cc:1428] Setting key 1027cf8c5d3a4e5e272ede16d29fbdcb
I20260924 15:23:22.195634 8536 raft_consensus.cc:348] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605a64c7f574743ace48b98d71248c1" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40677 } }
I20260924 15:23:22.195811 8536 raft_consensus.cc:374] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:22.195850 8536 raft_consensus.cc:729] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c605a64c7f574743ace48b98d71248c1, State: Initialized, Role: FOLLOWER
I20260924 15:23:22.196043 8536 consensus_queue.cc:260] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605a64c7f574743ace48b98d71248c1" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40677 } }
I20260924 15:23:22.196137 8536 raft_consensus.cc:388] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:22.196168 8536 raft_consensus.cc:482] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:22.196219 8536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:22.196839 8536 raft_consensus.cc:504] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605a64c7f574743ace48b98d71248c1" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40677 } }
I20260924 15:23:22.196947 8536 leader_election.cc:302] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c605a64c7f574743ace48b98d71248c1; no voters:
I20260924 15:23:22.197091 8536 leader_election.cc:288] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:22.197181 8541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:22.197324 8536 sys_catalog.cc:564] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:22.197362 8541 raft_consensus.cc:686] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [term 1 LEADER]: Becoming Leader. State: Replica: c605a64c7f574743ace48b98d71248c1, State: Running, Role: LEADER
I20260924 15:23:22.197572 8541 consensus_queue.cc:237] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605a64c7f574743ace48b98d71248c1" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40677 } }
I20260924 15:23:22.197942 8543 sys_catalog.cc:455] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c605a64c7f574743ace48b98d71248c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605a64c7f574743ace48b98d71248c1" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40677 } } }
I20260924 15:23:22.198261 8543 sys_catalog.cc:458] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:22.197927 8541 sys_catalog.cc:455] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c605a64c7f574743ace48b98d71248c1. Latest consensus state: current_term: 1 leader_uuid: "c605a64c7f574743ace48b98d71248c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c605a64c7f574743ace48b98d71248c1" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40677 } } }
I20260924 15:23:22.198550 8541 sys_catalog.cc:458] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:22.198484 8550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:22.199154 8550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:22.200731 8550 catalog_manager.cc:1347] Generated new cluster ID: 7c36629519324516baca9d149c8244d3
I20260924 15:23:22.200786 8550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:22.205737 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.195023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.4.78.126:40677)
0924 15:23:22.195247 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.195256 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.195300 (+ 44us) server_negotiation.cc:407] Connection header received
0924 15:23:22.195598 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.195606 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.195674 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.195819 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:22.196606 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.198103 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.199271 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.199930 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.200034 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.202512 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:22.202516 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:22.202517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:22.202528 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:22.204126 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.204494 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.204498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.204498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.204601 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.204805 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.204806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.204806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.205113 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:22.205281 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.205392 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.205530 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260924 15:23:22.218839 8550 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:22.219376 8550 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:22.222368 8550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1: Generated new TSK 0
I20260924 15:23:22.222668 8550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:22.254539 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40677
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:22.355973 8562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:22.356181 8562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:22.356206 8562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:22.357358 8562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:22.357398 8562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:22.357436 8562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:22.358929 8562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:22.359177 8562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:22.359421 8562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:22.364408 8567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:22.364393 8568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:22.364441 8562 server_base.cc:1034] running on GCE node
W20260924 15:23:22.364393 8570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:22.364859 8562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:22.365087 8562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:22.366246 8562 hybrid_clock.cc:648] HybridClock initialized: now 1790263402366235 us; error 29 us; skew 500 ppm
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:22.368721 8562 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:22.369046 8562 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:22.369094 8562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:22.369249 8562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:22.370304 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "a2dd365a703547638f13e1fd618429b5"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "219e45a01d7836f87272f7e75b8010ea"
server_key_iv: "1ad28cb6818a998d16534f33ca3bbf9f"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.370661 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "a2dd365a703547638f13e1fd618429b5"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "219e45a01d7836f87272f7e75b8010ea"
server_key_iv: "1ad28cb6818a998d16534f33ca3bbf9f"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.371902 8562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:22.372579 8576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.372735 8562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:22.372833 8562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "a2dd365a703547638f13e1fd618429b5"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "219e45a01d7836f87272f7e75b8010ea"
server_key_iv: "1ad28cb6818a998d16534f33ca3bbf9f"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.372900 8562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:22.387836 8562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:22.389614 8562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:22.389775 8562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:22.390018 8562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:22.390342 8562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:22.390399 8562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.390434 8562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:22.390460 8562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.395960 8562 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:41053
I20260924 15:23:22.396027 8689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:41053 every 8 connection(s)
I20260924 15:23:22.396296 8562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:22.399641 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 8562
I20260924 15:23:22.399731 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:22.399924 4409 external_mini_cluster.cc:1428] Setting key 0bb46f8a37521cd25858ddcd71aa3ac0
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:22.406275 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.396988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:60105 (local address 127.4.78.126:40677)
0924 15:23:22.397071 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.397074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.397249 (+ 175us) server_negotiation.cc:407] Connection header received
0924 15:23:22.397759 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.397760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.397772 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.397942 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:22.398851 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.399391 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.400641 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.400972 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.401045 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.403460 (+ 2415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:22.403461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:22.403461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:22.403471 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:22.404809 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.405294 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.405295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.405295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.405357 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.405605 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.405606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.405612 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.405728 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:22.405803 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.406093 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.406188 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:23:22.406443 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.396954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40677 (local address 127.4.78.65:60105)
0924 15:23:22.397135 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.397171 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:22.397673 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:22.398015 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:22.398018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:22.398267 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:22.398773 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.398778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.399488 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.399489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.400533 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.400536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.401047 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.401048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.401150 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.401644 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:22.401652 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:22.403362 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:22.404904 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:22.404910 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:22.404912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:22.405230 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:22.405424 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:22.405425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:22.405425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:22.405538 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:22.405793 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:22.405799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:22.405952 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 15:23:22.406057 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:23:22.406215 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260924 15:23:22.407145 8690 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40677
I20260924 15:23:22.407241 8690 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:22.407382 8690 heartbeater.cc:511] Master 127.4.78.126:40677 requested a full tablet report, sending...
I20260924 15:23:22.407966 8500 ts_manager.cc:194] Registered new tserver with Master: a2dd365a703547638f13e1fd618429b5 (127.4.78.65:41053)
I20260924 15:23:22.408936 8500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:60105
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:22.426259 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40677
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:22.506924 8695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:22.507113 8695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:22.507136 8695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:22.508288 8695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:22.508327 8695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:22.508374 8695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:22.509886 8695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:22.510124 8695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:22.510344 8695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:22.514849 8701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:22.514904 8703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:22.514849 8700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:22.515012 8695 server_base.cc:1034] running on GCE node
I20260924 15:23:22.515229 8695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:22.515442 8695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:22.516615 8695 hybrid_clock.cc:648] HybridClock initialized: now 1790263402516596 us; error 29 us; skew 500 ppm
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:22.519165 8695 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:22.519505 8695 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:22.519568 8695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:22.519716 8695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:22.520936 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "df8dbca243f846d09ca6cc1acf8d7219"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "719e4cd841da0ea08bb4e3f0a62cdade"
server_key_iv: "050ad59895cdaf2df14bb8bfe776f2e5"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.521356 8695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "df8dbca243f846d09ca6cc1acf8d7219"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "719e4cd841da0ea08bb4e3f0a62cdade"
server_key_iv: "050ad59895cdaf2df14bb8bfe776f2e5"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.523130 8695 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260924 15:23:22.524001 8709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.524210 8695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:22.524312 8695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "df8dbca243f846d09ca6cc1acf8d7219"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "719e4cd841da0ea08bb4e3f0a62cdade"
server_key_iv: "050ad59895cdaf2df14bb8bfe776f2e5"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.524387 8695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:22.534770 8695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:22.536870 8695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:22.536978 8695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:22.537144 8695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:22.537384 8695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:22.537412 8695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.537432 8695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:22.537444 8695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.543368 8695 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:35817
I20260924 15:23:22.543406 8822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:35817 every 8 connection(s)
I20260924 15:23:22.543650 8695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:22.550648 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 8695
I20260924 15:23:22.550769 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:22.550963 4409 external_mini_cluster.cc:1428] Setting key 5bb466f26bf0248aa19ec9da8c06f0f4
I20260924 15:23:22.552582 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.544252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:47685 (local address 127.4.78.126:40677)
0924 15:23:22.544297 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.544299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.544526 (+ 227us) server_negotiation.cc:407] Connection header received
0924 15:23:22.545009 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.545010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.545021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.545074 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:22.546015 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.546510 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.547487 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.547679 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.547729 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.549641 (+ 1912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:22.549642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:22.549643 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:22.549649 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:22.551141 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.551626 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.551627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.551627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.551686 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.551928 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.551929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.551929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.552056 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:22.552133 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.552396 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.552479 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:23:22.552763 8825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.544216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40677 (local address 127.4.78.66:47685)
0924 15:23:22.544373 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.544409 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:22.544917 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:22.545133 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:22.545136 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:22.545386 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:22.545935 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.545941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.546595 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.546596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.547415 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.547417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.547746 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.547747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.547841 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.548250 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:22.548258 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:22.549554 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:22.551228 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:22.551235 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:22.551238 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:22.551527 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:22.551757 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:22.551758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:22.551759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:22.551862 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:22.552132 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:22.552140 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:22.552271 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 15:23:22.552375 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 15:23:22.552526 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260924 15:23:22.553197 8823 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40677
I20260924 15:23:22.553301 8823 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:22.553439 8823 heartbeater.cc:511] Master 127.4.78.126:40677 requested a full tablet report, sending...
I20260924 15:23:22.553795 8500 ts_manager.cc:194] Registered new tserver with Master: df8dbca243f846d09ca6cc1acf8d7219 (127.4.78.66:35817)
I20260924 15:23:22.554206 8500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:47685
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:22.574388 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40677
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:22.652667 8828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:22.652849 8828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:22.652872 8828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:22.654021 8828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:22.654059 8828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:22.654109 8828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:22.655541 8828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:22.655761 8828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:22.655972 8828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:22.660610 8833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:22.660620 8834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:22.660650 8828 server_base.cc:1034] running on GCE node
W20260924 15:23:22.660624 8836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:22.660989 8828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:22.661211 8828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:22.662362 8828 hybrid_clock.cc:648] HybridClock initialized: now 1790263402662354 us; error 26 us; skew 500 ppm
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:22.664556 8828 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:22.664921 8828 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:22.664983 8828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:22.665112 8828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:22.666154 8828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "c34dac9e7b3147b39df6278d84dd880c"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "9050dac379e71fe3ae3cd512e4d100c8"
server_key_iv: "e48bc2f8aaae7bcfba8762fd6bc542eb"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.666492 8828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "c34dac9e7b3147b39df6278d84dd880c"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "9050dac379e71fe3ae3cd512e4d100c8"
server_key_iv: "e48bc2f8aaae7bcfba8762fd6bc542eb"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.667701 8828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:22.668377 8842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.668534 8828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:22.668627 8828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "c34dac9e7b3147b39df6278d84dd880c"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "9050dac379e71fe3ae3cd512e4d100c8"
server_key_iv: "e48bc2f8aaae7bcfba8762fd6bc542eb"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.668700 8828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:22.684260 8828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:22.685982 8828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:22.686183 8828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:22.686393 8828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:22.686685 8828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:22.686733 8828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.686774 8828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:22.686800 8828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.692474 8828 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:40513
I20260924 15:23:22.692534 8955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:40513 every 8 connection(s)
I20260924 15:23:22.692782 8828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:22.698459 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 8828
I20260924 15:23:22.698593 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
Sep 24 15:23:22 dist-test-slave-l766 krb5kdc[8456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263402, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:22.698828 4409 external_mini_cluster.cc:1428] Setting key ba7af0e953cd35c98416ff38cefb2ae2
I20260924 15:23:22.701656 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.693466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:34789 (local address 127.4.78.126:40677)
0924 15:23:22.693547 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.693550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.693757 (+ 207us) server_negotiation.cc:407] Connection header received
0924 15:23:22.694284 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.694284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.694294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.694333 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:22.695211 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.695694 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.696646 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.696816 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.696864 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.699278 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:22.699279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:22.699279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:22.699286 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:22.700586 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.700930 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.700931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.700931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.700978 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.701172 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.701172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.701173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.701268 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:22.701316 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.701536 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.701594 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 15:23:22.701843 8958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.693447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40677 (local address 127.4.78.67:34789)
0924 15:23:22.693647 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.693681 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:22.694189 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:22.694392 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:22.694395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:22.694629 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:22.695105 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.695110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.695782 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.695783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.696577 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.696579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.696883 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.696883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.696995 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.697571 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:22.697579 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:22.699184 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:22.700651 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:22.700657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:22.700658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:22.700868 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:22.701032 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:22.701032 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:22.701033 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:22.701123 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:22.701318 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:22.701321 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:22.701433 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 15:23:22.701513 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:23:22.701626 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":128,"threads_started":1}
I20260924 15:23:22.702291 8956 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40677
I20260924 15:23:22.702389 8956 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:22.702536 8956 heartbeater.cc:511] Master 127.4.78.126:40677 requested a full tablet report, sending...
I20260924 15:23:22.702847 8500 ts_manager.cc:194] Registered new tserver with Master: c34dac9e7b3147b39df6278d84dd880c (127.4.78.67:40513)
I20260924 15:23:22.703222 8500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:34789
I20260924 15:23:22.710306 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:23:22.717492 8540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.711512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.4.78.126:40677)
0924 15:23:22.711556 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.711558 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.711672 (+ 114us) server_negotiation.cc:407] Connection header received
0924 15:23:22.711747 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.711748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.711769 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.711824 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:22.712368 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.712840 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.713554 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.713752 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.713850 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.714980 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:22.714981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:22.714982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:22.714988 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:22.716330 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.716673 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.716674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.716674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.716728 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:22.716932 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:22.716933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:22.716934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:22.717078 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:22.717187 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.717317 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.717394 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:23:22.719064 8500 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:22.720574 8500 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:22.726953 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.723009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48752 (local address 127.4.78.66:35817)
0924 15:23:22.723223 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.723227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.723238 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:23:22.723281 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.723282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.723296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.723373 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:22.723542 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.724694 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.726028 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.726525 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.726607 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.726668 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.726828 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.726866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260924 15:23:22.726940 8965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.722888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:35817 (local address 127.0.0.1:48752)
0924 15:23:22.723175 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.723184 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:22.723224 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:22.723379 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:22.723381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:22.723384 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:22.723450 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.723452 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.724812 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.724813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.725947 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.725949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.726608 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.726609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.726702 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.726717 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 15:23:22.726811 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 15:23:22.726868 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":63,"thread_start_us":38,"threads_started":1}
I20260924 15:23:22.727137 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.722801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.4.78.67:40513)
0924 15:23:22.723114 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.723119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.723130 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:23:22.723277 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.723279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.723296 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.723373 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:22.723624 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.724345 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.725872 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.726593 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.726665 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.726720 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.726982 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.727031 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":109,"threads_started":1}
I20260924 15:23:22.727171 8967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.722692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40513 (local address 127.0.0.1:55616)
0924 15:23:22.723079 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.723089 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:22.723154 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:22.723490 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:22.723491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:22.723493 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:22.723543 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.723545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.724448 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.724449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.725769 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.725771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.726761 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.726762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.726830 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.726833 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:22.726882 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:23:22.726923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":55,"threads_started":1}
I20260924 15:23:22.727396 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.722865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39904 (local address 127.4.78.65:41053)
0924 15:23:22.723030 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.723034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.723239 (+ 205us) server_negotiation.cc:407] Connection header received
0924 15:23:22.723287 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.723290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.723305 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.723373 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:22.723583 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.724677 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.726049 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.726745 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.726814 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.726865 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.727156 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.727319 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260924 15:23:22.727632 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.722666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:41053 (local address 127.0.0.1:39904)
0924 15:23:22.723039 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.723083 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:22.723155 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:22.723459 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:22.723460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:22.723462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:22.723507 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.723508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.724829 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.724830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.725962 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.725965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.726897 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.726898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.726947 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.726950 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:22.727526 (+ 576us) client_negotiation.cc:240] Negotiation successful
0924 15:23:22.727570 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":3,"thread_start_us":55,"threads_started":1}
I20260924 15:23:22.727859 8757 tablet_service.cc:1467] Processing CreateTablet for tablet d8045c654f35498e958fb56cfdbb1453 (DEFAULT_TABLE table=test-table [id=47d5a38eaace46fbac08aa676272ee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:22.727895 8890 tablet_service.cc:1467] Processing CreateTablet for tablet d8045c654f35498e958fb56cfdbb1453 (DEFAULT_TABLE table=test-table [id=47d5a38eaace46fbac08aa676272ee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:22.728256 8757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8045c654f35498e958fb56cfdbb1453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:22.728255 8890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8045c654f35498e958fb56cfdbb1453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:22.728609 8624 tablet_service.cc:1467] Processing CreateTablet for tablet d8045c654f35498e958fb56cfdbb1453 (DEFAULT_TABLE table=test-table [id=47d5a38eaace46fbac08aa676272ee62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:22.728899 8624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8045c654f35498e958fb56cfdbb1453. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:22.731267 8970 tablet_bootstrap.cc:492] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: Bootstrap starting.
I20260924 15:23:22.731549 8971 tablet_bootstrap.cc:492] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Bootstrap starting.
I20260924 15:23:22.731972 8970 tablet_bootstrap.cc:654] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:22.732050 8972 tablet_bootstrap.cc:492] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5: Bootstrap starting.
I20260924 15:23:22.732309 8971 tablet_bootstrap.cc:654] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:22.732625 8970 log.cc:824] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:22.732671 8972 tablet_bootstrap.cc:654] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:22.732924 8971 log.cc:824] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:22.733268 8972 log.cc:824] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:22.733457 8970 tablet_bootstrap.cc:492] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: No bootstrap required, opened a new log
I20260924 15:23:22.733528 8970 ts_tablet_manager.cc:1397] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:22.733736 8971 tablet_bootstrap.cc:492] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: No bootstrap required, opened a new log
I20260924 15:23:22.733839 8971 ts_tablet_manager.cc:1397] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:22.734102 8972 tablet_bootstrap.cc:492] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5: No bootstrap required, opened a new log
I20260924 15:23:22.734207 8972 ts_tablet_manager.cc:1397] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:22.735430 8971 raft_consensus.cc:348] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.735560 8971 raft_consensus.cc:374] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:22.735584 8971 raft_consensus.cc:729] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df8dbca243f846d09ca6cc1acf8d7219, State: Initialized, Role: FOLLOWER
I20260924 15:23:22.735549 8970 raft_consensus.cc:348] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.735726 8970 raft_consensus.cc:374] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:22.735725 8971 consensus_queue.cc:260] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.735775 8970 raft_consensus.cc:729] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34dac9e7b3147b39df6278d84dd880c, State: Initialized, Role: FOLLOWER
I20260924 15:23:22.735903 8970 consensus_queue.cc:260] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.735991 8823 heartbeater.cc:503] Master 127.4.78.126:40677 was elected leader, sending a full tablet report...
I20260924 15:23:22.736001 8971 ts_tablet_manager.cc:1428] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:23:22.736172 8970 ts_tablet_manager.cc:1428] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:23:22.736296 8956 heartbeater.cc:503] Master 127.4.78.126:40677 was elected leader, sending a full tablet report...
I20260924 15:23:22.736554 8972 raft_consensus.cc:348] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.736698 8972 raft_consensus.cc:374] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:22.736742 8972 raft_consensus.cc:729] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2dd365a703547638f13e1fd618429b5, State: Initialized, Role: FOLLOWER
I20260924 15:23:22.736861 8972 consensus_queue.cc:260] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.737143 8972 ts_tablet_manager.cc:1428] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:23:22.737172 8690 heartbeater.cc:503] Master 127.4.78.126:40677 was elected leader, sending a full tablet report...
W20260924 15:23:22.794164 8824 tablet.cc:2369] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:22.807349 8978 raft_consensus.cc:482] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:22.807464 8978 raft_consensus.cc:504] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.807782 8978 leader_election.cc:288] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers df8dbca243f846d09ca6cc1acf8d7219 (127.4.78.66:35817), c34dac9e7b3147b39df6278d84dd880c (127.4.78.67:40513)
I20260924 15:23:22.811584 8969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.808019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:59103 (local address 127.4.78.66:35817)
0924 15:23:22.808075 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.808077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.808088 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:23:22.808121 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.808122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.808134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.808219 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:22.808366 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.809130 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.810631 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.811205 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.811303 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.811313 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.811471 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.811513 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:23:22.811584 8966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.808064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:50135 (local address 127.4.78.67:40513)
0924 15:23:22.808107 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.808109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.808214 (+ 105us) server_negotiation.cc:407] Connection header received
0924 15:23:22.808287 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.808289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.808306 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.808372 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:22.808510 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.809199 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.810630 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.811205 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.811284 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.811296 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.811472 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.811513 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:23:22.811590 8692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.807948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:35817 (local address 127.4.78.65:59103)
0924 15:23:22.807992 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.808004 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:22.808049 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:22.808237 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:22.808238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:22.808240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:22.808299 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.808301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.809209 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.809209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.810539 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.810541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.811299 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.811300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.811372 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.811374 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:23:22.811421 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:23:22.811477 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":43}
I20260924 15:23:22.811600 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.807948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40513 (local address 127.4.78.65:50135)
0924 15:23:22.808144 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.808152 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:22.808214 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:22.808383 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:22.808384 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:22.808385 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:22.808446 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.808447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.809264 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.809265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.810533 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:22.810538 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.811284 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.811285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:22.811360 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.811364 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:22.811455 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 15:23:22.811487 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":73,"thread_start_us":108,"threads_started":1}
I20260924 15:23:22.811945 8777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8045c654f35498e958fb56cfdbb1453" candidate_uuid: "a2dd365a703547638f13e1fd618429b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8dbca243f846d09ca6cc1acf8d7219" is_pre_election: true
I20260924 15:23:22.811993 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8045c654f35498e958fb56cfdbb1453" candidate_uuid: "a2dd365a703547638f13e1fd618429b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34dac9e7b3147b39df6278d84dd880c" is_pre_election: true
I20260924 15:23:22.812119 8777 raft_consensus.cc:2454] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2dd365a703547638f13e1fd618429b5 in term 0.
I20260924 15:23:22.812117 8910 raft_consensus.cc:2454] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2dd365a703547638f13e1fd618429b5 in term 0.
I20260924 15:23:22.812358 8578 leader_election.cc:302] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2dd365a703547638f13e1fd618429b5, c34dac9e7b3147b39df6278d84dd880c; no voters:
I20260924 15:23:22.812487 8978 raft_consensus.cc:2781] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:22.812542 8978 raft_consensus.cc:482] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:22.812567 8978 raft_consensus.cc:3037] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:22.813220 8978 raft_consensus.cc:504] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.813356 8978 leader_election.cc:288] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [CANDIDATE]: Term 1 election: Requested vote from peers df8dbca243f846d09ca6cc1acf8d7219 (127.4.78.66:35817), c34dac9e7b3147b39df6278d84dd880c (127.4.78.67:40513)
I20260924 15:23:22.813521 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8045c654f35498e958fb56cfdbb1453" candidate_uuid: "a2dd365a703547638f13e1fd618429b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34dac9e7b3147b39df6278d84dd880c"
I20260924 15:23:22.813517 8777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8045c654f35498e958fb56cfdbb1453" candidate_uuid: "a2dd365a703547638f13e1fd618429b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "df8dbca243f846d09ca6cc1acf8d7219"
I20260924 15:23:22.813597 8777 raft_consensus.cc:3037] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:22.813597 8910 raft_consensus.cc:3037] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:22.814498 8777 raft_consensus.cc:2454] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2dd365a703547638f13e1fd618429b5 in term 1.
I20260924 15:23:22.814498 8910 raft_consensus.cc:2454] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2dd365a703547638f13e1fd618429b5 in term 1.
I20260924 15:23:22.814659 8578 leader_election.cc:302] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2dd365a703547638f13e1fd618429b5, c34dac9e7b3147b39df6278d84dd880c; no voters:
I20260924 15:23:22.814787 8978 raft_consensus.cc:2781] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:22.814952 8978 raft_consensus.cc:686] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [term 1 LEADER]: Becoming Leader. State: Replica: a2dd365a703547638f13e1fd618429b5, State: Running, Role: LEADER
I20260924 15:23:22.815061 8978 consensus_queue.cc:237] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } }
I20260924 15:23:22.815924 8499 catalog_manager.cc:5495] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2dd365a703547638f13e1fd618429b5 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "a2dd365a703547638f13e1fd618429b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2dd365a703547638f13e1fd618429b5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 } health_report { overall_health: UNKNOWN } } }
I20260924 15:23:22.841908 8968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.839480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.4.78.65:41053)
0924 15:23:22.839531 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.839533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.839543 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:22.839590 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.839591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.839613 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.839693 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:22.839844 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.840319 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.841073 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:22.841285 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:22.841358 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:22.841531 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:22.841636 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:22.841719 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.841763 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.841787 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:23:22.843612 8777 raft_consensus.cc:1261] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 1 FOLLOWER]: Refusing update from remote peer a2dd365a703547638f13e1fd618429b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:22.843647 8910 raft_consensus.cc:1261] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 1 FOLLOWER]: Refusing update from remote peer a2dd365a703547638f13e1fd618429b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:22.843951 8978 consensus_queue.cc:1035] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "df8dbca243f846d09ca6cc1acf8d7219" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:22.844080 8980 consensus_queue.cc:1035] T d8045c654f35498e958fb56cfdbb1453 P a2dd365a703547638f13e1fd618429b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34dac9e7b3147b39df6278d84dd880c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:22.846550 8983 mvcc.cc:205] Tried to move back new op lower bound from 7332918898044911616 to 7332918897931149312. Current Snapshot: MvccSnapshot[applied={T|T < 7332918898044911616}]
I20260924 15:23:22.846691 8985 mvcc.cc:205] Tried to move back new op lower bound from 7332918898044911616 to 7332918897931149312. Current Snapshot: MvccSnapshot[applied={T|T < 7332918898044911616}]
I20260924 15:23:22.848683 8500 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:22.848778 8500 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:22.849530 8500 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c605a64c7f574743ace48b98d71248c1: Sending DeleteTablet for 3 replicas of tablet d8045c654f35498e958fb56cfdbb1453
I20260924 15:23:22.849844 8624 tablet_service.cc:1514] Processing DeleteTablet for tablet d8045c654f35498e958fb56cfdbb1453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:22 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:39904
I20260924 15:23:22.849900 8757 tablet_service.cc:1514] Processing DeleteTablet for tablet d8045c654f35498e958fb56cfdbb1453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:22 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:48752
I20260924 15:23:22.849901 8890 tablet_service.cc:1514] Processing DeleteTablet for tablet d8045c654f35498e958fb56cfdbb1453 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:22 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:55616
I20260924 15:23:22.850085 8994 tablet_replica.cc:330] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: stopping tablet replica
I20260924 15:23:22.850174 8994 raft_consensus.cc:2229] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:22.850209 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 8562
I20260924 15:23:22.850813 8994 raft_consensus.cc:2258] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:22.850910 8995 tablet_replica.cc:330] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: stopping tablet replica
I20260924 15:23:22.851006 8995 raft_consensus.cc:2229] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:22.851099 8995 raft_consensus.cc:2258] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:22.851399 8994 ts_tablet_manager.cc:1905] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:22.852629 8995 ts_tablet_manager.cc:1905] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:22.853351 8994 ts_tablet_manager.cc:1918] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:22.853431 8994 log.cc:1196] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/d8045c654f35498e958fb56cfdbb1453
I20260924 15:23:22.853717 8994 ts_tablet_manager.cc:1939] T d8045c654f35498e958fb56cfdbb1453 P df8dbca243f846d09ca6cc1acf8d7219: Deleting consensus metadata
I20260924 15:23:22.853981 8995 ts_tablet_manager.cc:1918] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:22.854038 8995 log.cc:1196] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/d8045c654f35498e958fb56cfdbb1453
I20260924 15:23:22.854121 8484 catalog_manager.cc:4841] TS df8dbca243f846d09ca6cc1acf8d7219 (127.4.78.66:35817): tablet d8045c654f35498e958fb56cfdbb1453 (table test-table [id=47d5a38eaace46fbac08aa676272ee62]) successfully deleted
I20260924 15:23:22.854315 8995 ts_tablet_manager.cc:1939] T d8045c654f35498e958fb56cfdbb1453 P c34dac9e7b3147b39df6278d84dd880c: Deleting consensus metadata
I20260924 15:23:22.854655 8485 catalog_manager.cc:4841] TS c34dac9e7b3147b39df6278d84dd880c (127.4.78.67:40513): tablet d8045c654f35498e958fb56cfdbb1453 (table test-table [id=47d5a38eaace46fbac08aa676272ee62]) successfully deleted
W20260924 15:23:22.856413 8484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:41053 (error 108)
I20260924 15:23:22.856716 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 8695
I20260924 15:23:22.856873 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.856718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:41053 (local address 127.0.0.1:39936)
0924 15:23:22.856794 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:22.856837 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:41053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:23:22.857856 8484 catalog_manager.cc:4584] TS a2dd365a703547638f13e1fd618429b5 (127.4.78.65:41053): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d8045c654f35498e958fb56cfdbb1453: Network error: Client connection negotiation failed: client connection to 127.4.78.65:41053: connect: Connection refused (error 111)
I20260924 15:23:22.861276 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 8828
I20260924 15:23:22.865495 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 8469
2026-09-24T15:23:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1932 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T15:23:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:22Z Disabled control of system clock
I20260924 15:23:22.892282 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39831
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:23:22.971487 9000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:22.971676 9000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:22.971698 9000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:22.972761 9000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:22.972795 9000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:22.972807 9000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:22.972820 9000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:22.972832 9000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:22.974080 9000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39831
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:22.974253 9000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:22.974476 9000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:22.978915 9005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:22.979048 9000 server_base.cc:1034] running on GCE node
W20260924 15:23:22.978950 9006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:22.978940 9008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:22.979410 9000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:22.979692 9000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:22.980850 9000 hybrid_clock.cc:648] HybridClock initialized: now 1790263402980837 us; error 24 us; skew 500 ppm
I20260924 15:23:22.982151 9000 webserver.cc:460] Webserver started at http://127.4.78.126:33941/ using document root <none> and password file <none>
I20260924 15:23:22.982390 9000 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:22.982461 9000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:22.982596 9000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:22.983615 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "944323237a804bc4b25ab1fd82562efb"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "b01fb501006df2d9c691f497c6efd610"
server_key_iv: "3174b8091c139b1bbb6ca5033d9e9d63"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.983982 9000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "944323237a804bc4b25ab1fd82562efb"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "b01fb501006df2d9c691f497c6efd610"
server_key_iv: "3174b8091c139b1bbb6ca5033d9e9d63"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.985235 9000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:22.986963 9014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:22.987131 9000 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:22.987226 9000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "944323237a804bc4b25ab1fd82562efb"
format_stamp: "Formatted at 2026-09-24 15:23:22 on dist-test-slave-l766"
server_key: "b01fb501006df2d9c691f497c6efd610"
server_key_iv: "3174b8091c139b1bbb6ca5033d9e9d63"
server_key_version: "encryptionkey@0"
I20260924 15:23:22.987314 9000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:22.990626 9000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:22.990880 9000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:22.991024 9000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:22.994867 9000 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:39831
I20260924 15:23:22.994897 9066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:39831 every 8 connection(s)
I20260924 15:23:22.995199 9000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:22.995796 9067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:22.995965 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9000
I20260924 15:23:22.996061 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:22.996265 4409 external_mini_cluster.cc:1428] Setting key 9a359f2b2a47d8f3ecbbdebdecc5fc3a
I20260924 15:23:22.998795 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:22.996770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.4.78.126:39831)
0924 15:23:22.996976 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:22.996983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:22.997005 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:23:22.997431 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:22.997437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:22.997474 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:22.997614 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:22.998396 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:22.998399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:22.998399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:22.998403 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:22.998438 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:22.998448 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:22.998521 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:22.998594 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 15:23:22.998655 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
I20260924 15:23:22.999650 9067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb: Bootstrap starting.
I20260924 15:23:23.000231 9067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:23.000546 9067 log.cc:824] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:23.001245 9067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb: No bootstrap required, opened a new log
I20260924 15:23:23.002609 9067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944323237a804bc4b25ab1fd82562efb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39831 } }
I20260924 15:23:23.002754 9067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:23.002815 9067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944323237a804bc4b25ab1fd82562efb, State: Initialized, Role: FOLLOWER
I20260924 15:23:23.002954 9067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944323237a804bc4b25ab1fd82562efb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39831 } }
I20260924 15:23:23.003048 9067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:23.003093 9067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:23.003144 9067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:23.003654 9067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944323237a804bc4b25ab1fd82562efb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39831 } }
I20260924 15:23:23.003765 9067 leader_election.cc:302] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 944323237a804bc4b25ab1fd82562efb; no voters:
I20260924 15:23:23.003928 9067 leader_election.cc:288] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:23.004173 9067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:23.004181 9072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:23.004325 9072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [term 1 LEADER]: Becoming Leader. State: Replica: 944323237a804bc4b25ab1fd82562efb, State: Running, Role: LEADER
I20260924 15:23:23.004438 9072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944323237a804bc4b25ab1fd82562efb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39831 } }
I20260924 15:23:23.004793 9072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 944323237a804bc4b25ab1fd82562efb. Latest consensus state: current_term: 1 leader_uuid: "944323237a804bc4b25ab1fd82562efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944323237a804bc4b25ab1fd82562efb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39831 } } }
I20260924 15:23:23.004886 9072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:23.004803 9073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "944323237a804bc4b25ab1fd82562efb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "944323237a804bc4b25ab1fd82562efb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39831 } } }
I20260924 15:23:23.005012 9073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:23.005113 9081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:23.005620 9081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:23.007076 9081 catalog_manager.cc:1347] Generated new cluster ID: 391a75d1b45b4e99813a9841e5ab0b57
I20260924 15:23:23.007129 9081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:23.013741 9081 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:23.013934 9081 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:23.019822 9081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb: Generated new TSK 0
I20260924 15:23:23.019990 9081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:23:23.025436 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39831
--builtin_ntp_servers=127.4.78.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:23:23.106323 9091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:23.106530 9091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:23.106567 9091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:23.107710 9091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:23.107775 9091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:23.107820 9091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:23.109165 9091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:23.109412 9091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:23.109726 9091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:23.114373 9096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:23.114396 9099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:23.114363 9097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:23.114571 9091 server_base.cc:1034] running on GCE node
I20260924 15:23:23.114832 9091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:23.115092 9091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:23.116250 9091 hybrid_clock.cc:648] HybridClock initialized: now 1790263403116233 us; error 33 us; skew 500 ppm
I20260924 15:23:23.117509 9091 webserver.cc:460] Webserver started at http://127.4.78.65:38721/ using document root <none> and password file <none>
I20260924 15:23:23.117770 9091 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:23.117857 9091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:23.117993 9091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:23.119027 9091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "4d93b3d72b624006b7ac8293e0aa11e4"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "42b7adea8ad8cdc69955be9b4e322b2a"
server_key_iv: "dddb82a61788f293929faab9713f79b6"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.119386 9091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "4d93b3d72b624006b7ac8293e0aa11e4"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "42b7adea8ad8cdc69955be9b4e322b2a"
server_key_iv: "dddb82a61788f293929faab9713f79b6"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.120752 9091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:23.121523 9105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.121709 9091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:23.121796 9091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "4d93b3d72b624006b7ac8293e0aa11e4"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "42b7adea8ad8cdc69955be9b4e322b2a"
server_key_iv: "dddb82a61788f293929faab9713f79b6"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.121903 9091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:23.125131 9091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:23.125404 9091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:23.125542 9091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:23.125778 9091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:23.126140 9091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:23.126196 9091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.126237 9091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:23.126256 9091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.132452 9091 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:33953
I20260924 15:23:23.132490 9218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:33953 every 8 connection(s)
I20260924 15:23:23.132802 9091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:23.135598 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.133517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:38147 (local address 127.4.78.126:39831)
0924 15:23:23.133576 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.133578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.133797 (+ 219us) server_negotiation.cc:407] Connection header received
0924 15:23:23.134323 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.134324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.134330 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.134389 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.135413 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.135415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.135415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.135419 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.135433 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.135437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.135483 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.135529 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.135540 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:23:23.135851 9221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.133509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39831 (local address 127.4.78.65:38147)
0924 15:23:23.133675 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.133712 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.134229 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.134438 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.134441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.134449 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:23.135278 (+ 829us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:23.135279 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:23.135306 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:23.135333 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:23.135481 (+ 148us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:23.135484 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:23.135486 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:23:23.135534 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:23:23.135580 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260924 15:23:23.136327 9219 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39831
I20260924 15:23:23.136405 9219 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:23.136514 9219 heartbeater.cc:511] Master 127.4.78.126:39831 requested a full tablet report, sending...
I20260924 15:23:23.136947 9031 ts_manager.cc:194] Registered new tserver with Master: 4d93b3d72b624006b7ac8293e0aa11e4 (127.4.78.65:33953)
I20260924 15:23:23.140108 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9091
I20260924 15:23:23.140209 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:23.140416 4409 external_mini_cluster.cc:1428] Setting key 689d87c0a0f2e7ecb37f94b164180100
I20260924 15:23:23.141235 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39831
--builtin_ntp_servers=127.4.78.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:23:23.223611 9222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:23.223814 9222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:23.223835 9222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:23.225147 9222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:23.225185 9222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:23.225271 9222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:23.226734 9222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:23.226969 9222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:23.227247 9222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:23.232043 9227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:23.232028 9228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:23.232021 9230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:23.232215 9222 server_base.cc:1034] running on GCE node
I20260924 15:23:23.232450 9222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:23.232693 9222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:23.233873 9222 hybrid_clock.cc:648] HybridClock initialized: now 1790263403233854 us; error 29 us; skew 500 ppm
I20260924 15:23:23.235044 9222 webserver.cc:460] Webserver started at http://127.4.78.66:39273/ using document root <none> and password file <none>
I20260924 15:23:23.235285 9222 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:23.235330 9222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:23.235471 9222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:23.236521 9222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "4064f9851ec34f4ab8d9fe7788987f3c"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "5644f07dedc1eb898c925896323f17c0"
server_key_iv: "0153319b9effdb0738b32248293112fd"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.236872 9222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "4064f9851ec34f4ab8d9fe7788987f3c"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "5644f07dedc1eb898c925896323f17c0"
server_key_iv: "0153319b9effdb0738b32248293112fd"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.238225 9222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:23.238963 9236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.239120 9222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:23.239192 9222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "4064f9851ec34f4ab8d9fe7788987f3c"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "5644f07dedc1eb898c925896323f17c0"
server_key_iv: "0153319b9effdb0738b32248293112fd"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.239279 9222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:23.242604 9222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:23.242852 9222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:23.242981 9222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:23.243188 9222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:23.243481 9222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:23.243513 9222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.243564 9222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:23.243592 9222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.249224 9222 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:43379
I20260924 15:23:23.249290 9350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:43379 every 8 connection(s)
I20260924 15:23:23.249591 9222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:23.252451 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.250267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:57811 (local address 127.4.78.126:39831)
0924 15:23:23.250331 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.250333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.250572 (+ 239us) server_negotiation.cc:407] Connection header received
0924 15:23:23.251063 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.251065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.251072 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.251135 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.252191 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.252192 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.252192 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.252197 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.252211 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.252215 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.252273 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.252363 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.252378 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:23:23.252640 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.250270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39831 (local address 127.4.78.66:57811)
0924 15:23:23.250459 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.250490 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.250958 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.251208 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.251211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.251219 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:23.252049 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:23.252050 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:23.252075 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:23.252102 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:23.252278 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:23.252283 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:23.252286 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:23.252374 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:23:23.252433 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260924 15:23:23.253098 9351 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39831
I20260924 15:23:23.253185 9351 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:23.253326 9351 heartbeater.cc:511] Master 127.4.78.126:39831 requested a full tablet report, sending...
I20260924 15:23:23.253675 9031 ts_manager.cc:194] Registered new tserver with Master: 4064f9851ec34f4ab8d9fe7788987f3c (127.4.78.66:43379)
I20260924 15:23:23.255448 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9222
I20260924 15:23:23.255532 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:23.255668 4409 external_mini_cluster.cc:1428] Setting key 7c6eda57c7ebc1a3a6b872bc18153dea
I20260924 15:23:23.256434 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39831
--builtin_ntp_servers=127.4.78.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:23:23.335693 9354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:23.335887 9354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:23.335909 9354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:23.337073 9354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:23.337111 9354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:23.337164 9354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:23.338550 9354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39831
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:23.338783 9354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:23.339084 9354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:23.343955 9362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:23.343937 9360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:23.343971 9354 server_base.cc:1034] running on GCE node
W20260924 15:23:23.343926 9359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:23.344331 9354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:23.344574 9354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:23.345727 9354 hybrid_clock.cc:648] HybridClock initialized: now 1790263403345719 us; error 34 us; skew 500 ppm
I20260924 15:23:23.346899 9354 webserver.cc:460] Webserver started at http://127.4.78.67:36135/ using document root <none> and password file <none>
I20260924 15:23:23.347131 9354 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:23.347194 9354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:23.347368 9354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:23.348394 9354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "cd183621a15f4b3aa6951679a0ed130a"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "8548596d57b99d60bb63216e95bec7e6"
server_key_iv: "52cd604ae103f245ca7681c8b95a7ebe"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.348766 9354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "cd183621a15f4b3aa6951679a0ed130a"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "8548596d57b99d60bb63216e95bec7e6"
server_key_iv: "52cd604ae103f245ca7681c8b95a7ebe"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.350082 9354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:23.350844 9368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.351025 9354 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:23.351131 9354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "cd183621a15f4b3aa6951679a0ed130a"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "8548596d57b99d60bb63216e95bec7e6"
server_key_iv: "52cd604ae103f245ca7681c8b95a7ebe"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.351217 9354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:23.354609 9354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:23.354821 9354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:23.354912 9354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:23.355059 9354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:23.355289 9354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:23.355320 9354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.355340 9354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:23.355352 9354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.360734 9354 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:40119
I20260924 15:23:23.360848 9481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:40119 every 8 connection(s)
I20260924 15:23:23.361070 9354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:23.363741 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.361744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:41695 (local address 127.4.78.126:39831)
0924 15:23:23.361803 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.361805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.362019 (+ 214us) server_negotiation.cc:407] Connection header received
0924 15:23:23.362433 (+ 414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.362434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.362439 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.362480 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.363556 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.363557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.363557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.363561 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.363579 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.363583 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.363628 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.363680 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.363691 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:23:23.363991 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.361722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39831 (local address 127.4.78.67:41695)
0924 15:23:23.361902 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.361939 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.362338 (+ 399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.362529 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.362533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.362540 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:23.363393 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:23.363394 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:23.363419 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:23.363451 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:23.363632 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:23.363634 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:23.363636 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:23:23.363687 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:23:23.363731 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":102,"threads_started":1}
I20260924 15:23:23.364463 9482 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39831
I20260924 15:23:23.364537 9482 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:23.364637 9482 heartbeater.cc:511] Master 127.4.78.126:39831 requested a full tablet report, sending...
I20260924 15:23:23.364933 9031 ts_manager.cc:194] Registered new tserver with Master: cd183621a15f4b3aa6951679a0ed130a (127.4.78.67:40119)
I20260924 15:23:23.370746 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9354
I20260924 15:23:23.370847 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:23.371016 4409 external_mini_cluster.cc:1428] Setting key af6273477d93b74a91490b44bf94edcc
I20260924 15:23:23.371842 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:23:23.374272 9070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.373120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.4.78.126:39831)
0924 15:23:23.373192 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.373194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.373308 (+ 114us) server_negotiation.cc:407] Connection header received
0924 15:23:23.373378 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.373379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.373385 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.373452 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.374062 (+ 610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.374063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.374063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.374067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.374079 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.374083 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.374124 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.374188 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.374202 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:23:23.375658 9031 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:23.377143 9031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:23.380930 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.379267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.4.78.66:43379)
0924 15:23:23.379415 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.379420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.379559 (+ 139us) server_negotiation.cc:407] Connection header received
0924 15:23:23.379644 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.379650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.379660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.379729 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.380628 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.380634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.380635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.380638 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.380680 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.380700 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.380759 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.380838 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.380853 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 15:23:23.380932 9490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.379088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43379 (local address 127.0.0.1:35752)
0924 15:23:23.379461 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.379503 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.379566 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.380065 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.380066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.380068 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:23.380511 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:23.380512 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:23.380524 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:23.380540 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:23.380753 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:23.380761 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:23.380764 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:23.380833 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:23:23.380847 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":68,"threads_started":1}
I20260924 15:23:23.381065 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.379100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40119 (local address 127.0.0.1:57436)
0924 15:23:23.379657 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.379666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.379711 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.380178 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.380178 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.380181 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:23.380680 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:23.380680 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:23.380688 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:23.380701 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:23.380950 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:23.380951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:23.380951 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 15:23:23.380995 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 15:23:23.381007 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":78,"threads_started":1}
I20260924 15:23:23.381332 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.379329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.4.78.67:40119)
0924 15:23:23.379491 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.379495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.379712 (+ 217us) server_negotiation.cc:407] Connection header received
0924 15:23:23.379762 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.379765 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.379773 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.380156 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.380797 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.380803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.380804 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.380809 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.380850 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.380875 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.380926 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.381211 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.381229 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":68,"threads_started":1}
I20260924 15:23:23.381440 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.379085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:33953 (local address 127.0.0.1:33924)
0924 15:23:23.379461 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.379504 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.379568 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.380028 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.380034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.380039 (+ 5us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:23.381038 (+ 999us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:23.381038 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:23.381044 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:23.381053 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:23.381312 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:23.381313 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:23.381314 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:23:23.381365 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:23:23.381378 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":88,"threads_started":1}
I20260924 15:23:23.381754 9284 tablet_service.cc:1467] Processing CreateTablet for tablet 399eef3b77124dc28caaf7d60ae1608a (DEFAULT_TABLE table=test-table [id=a3de630d1e974355b4f490903da399b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:23.381980 9416 tablet_service.cc:1467] Processing CreateTablet for tablet 399eef3b77124dc28caaf7d60ae1608a (DEFAULT_TABLE table=test-table [id=a3de630d1e974355b4f490903da399b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:23.382148 9284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399eef3b77124dc28caaf7d60ae1608a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:23.382293 9416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399eef3b77124dc28caaf7d60ae1608a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:23.385360 9497 tablet_bootstrap.cc:492] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Bootstrap starting.
I20260924 15:23:23.385612 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.379254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33924 (local address 127.4.78.65:33953)
0924 15:23:23.379510 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.379514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.379582 (+ 68us) server_negotiation.cc:407] Connection header received
0924 15:23:23.379616 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.379618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.379629 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.379704 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.381136 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.381146 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.381146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.381151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.381186 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.381221 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.381286 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.385499 (+ 4213us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.385521 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":93,"threads_started":1}
I20260924 15:23:23.386059 9497 tablet_bootstrap.cc:654] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:23.386267 9153 tablet_service.cc:1467] Processing CreateTablet for tablet 399eef3b77124dc28caaf7d60ae1608a (DEFAULT_TABLE table=test-table [id=a3de630d1e974355b4f490903da399b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:23.386358 9497 log.cc:824] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:23.386554 9153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399eef3b77124dc28caaf7d60ae1608a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:23.386696 9496 tablet_bootstrap.cc:492] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Bootstrap starting.
I20260924 15:23:23.387226 9496 tablet_bootstrap.cc:654] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:23.387501 9496 log.cc:824] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:23.388057 9497 tablet_bootstrap.cc:492] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: No bootstrap required, opened a new log
I20260924 15:23:23.388134 9497 ts_tablet_manager.cc:1397] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:23:23.388386 9496 tablet_bootstrap.cc:492] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: No bootstrap required, opened a new log
I20260924 15:23:23.388461 9496 ts_tablet_manager.cc:1397] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:23.389472 9500 tablet_bootstrap.cc:492] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4: Bootstrap starting.
I20260924 15:23:23.390298 9500 tablet_bootstrap.cc:654] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:23.390486 9497 raft_consensus.cc:348] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.390671 9497 raft_consensus.cc:374] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:23.390720 9497 raft_consensus.cc:729] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd183621a15f4b3aa6951679a0ed130a, State: Initialized, Role: FOLLOWER
I20260924 15:23:23.390714 9500 log.cc:824] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:23.390851 9497 consensus_queue.cc:260] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.390864 9496 raft_consensus.cc:348] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.391021 9496 raft_consensus.cc:374] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:23.391062 9496 raft_consensus.cc:729] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4064f9851ec34f4ab8d9fe7788987f3c, State: Initialized, Role: FOLLOWER
I20260924 15:23:23.391134 9497 ts_tablet_manager.cc:1428] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 15:23:23.391184 9496 consensus_queue.cc:260] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.391258 9482 heartbeater.cc:503] Master 127.4.78.126:39831 was elected leader, sending a full tablet report...
I20260924 15:23:23.391443 9496 ts_tablet_manager.cc:1428] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:23:23.391453 9351 heartbeater.cc:503] Master 127.4.78.126:39831 was elected leader, sending a full tablet report...
I20260924 15:23:23.391922 9500 tablet_bootstrap.cc:492] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4: No bootstrap required, opened a new log
I20260924 15:23:23.391989 9500 ts_tablet_manager.cc:1397] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:23:23.393915 9500 raft_consensus.cc:348] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.394045 9500 raft_consensus.cc:374] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:23.394088 9500 raft_consensus.cc:729] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d93b3d72b624006b7ac8293e0aa11e4, State: Initialized, Role: FOLLOWER
I20260924 15:23:23.394199 9500 consensus_queue.cc:260] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.394467 9500 ts_tablet_manager.cc:1428] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:23:23.394522 9219 heartbeater.cc:503] Master 127.4.78.126:39831 was elected leader, sending a full tablet report...
W20260924 15:23:23.500277 9352 tablet.cc:2369] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:23.610625 9501 raft_consensus.cc:482] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:23.610782 9501 raft_consensus.cc:504] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.611163 9501 leader_election.cc:288] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d93b3d72b624006b7ac8293e0aa11e4 (127.4.78.65:33953), 4064f9851ec34f4ab8d9fe7788987f3c (127.4.78.66:43379)
W20260924 15:23:23.611599 9483 tablet.cc:2369] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:23.612480 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.611317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:33953 (local address 127.4.78.67:58673)
0924 15:23:23.611387 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.611399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.611450 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.611703 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.611704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.611706 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:23.612127 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:23.612128 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:23.612135 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:23.612145 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:23.612339 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:23.612340 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:23.612340 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 15:23:23.612392 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:23:23.612407 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 15:23:23.612613 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.611411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:58673 (local address 127.4.78.65:33953)
0924 15:23:23.611450 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.611453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.611461 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:23.611576 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.611578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.611586 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.611643 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.612246 (+ 603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.612247 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.612248 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.612252 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.612271 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.612277 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.612331 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.612511 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.612524 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:23.612665 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.611438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:48259 (local address 127.4.78.66:43379)
0924 15:23:23.611504 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.611506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.611638 (+ 132us) server_negotiation.cc:407] Connection header received
0924 15:23:23.611684 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.611686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.611692 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.611746 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.612409 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.612410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.612410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.612413 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.612427 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.612430 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.612470 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.612578 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.612591 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:23:23.612931 9505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.611329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43379 (local address 127.4.78.67:48259)
0924 15:23:23.611548 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.611566 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.611624 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.611799 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.611799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.611801 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:23.612320 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:23.612321 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:23.612327 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:23.612337 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:23.612482 (+ 145us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:23.612482 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:23.612483 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:23:23.612859 (+ 376us) client_negotiation.cc:240] Negotiation successful
0924 15:23:23.612873 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":117,"threads_started":1}
I20260924 15:23:23.612951 9173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399eef3b77124dc28caaf7d60ae1608a" candidate_uuid: "cd183621a15f4b3aa6951679a0ed130a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" is_pre_election: true
I20260924 15:23:23.613147 9173 raft_consensus.cc:2454] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd183621a15f4b3aa6951679a0ed130a in term 0.
I20260924 15:23:23.613344 9305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399eef3b77124dc28caaf7d60ae1608a" candidate_uuid: "cd183621a15f4b3aa6951679a0ed130a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" is_pre_election: true
I20260924 15:23:23.613436 9370 leader_election.cc:302] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d93b3d72b624006b7ac8293e0aa11e4, cd183621a15f4b3aa6951679a0ed130a; no voters:
I20260924 15:23:23.613477 9305 raft_consensus.cc:2454] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd183621a15f4b3aa6951679a0ed130a in term 0.
I20260924 15:23:23.613554 9501 raft_consensus.cc:2781] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:23.613621 9501 raft_consensus.cc:482] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:23.613657 9501 raft_consensus.cc:3037] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:23.614372 9501 raft_consensus.cc:504] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.614502 9501 leader_election.cc:288] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [CANDIDATE]: Term 1 election: Requested vote from peers 4d93b3d72b624006b7ac8293e0aa11e4 (127.4.78.65:33953), 4064f9851ec34f4ab8d9fe7788987f3c (127.4.78.66:43379)
I20260924 15:23:23.614646 9173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399eef3b77124dc28caaf7d60ae1608a" candidate_uuid: "cd183621a15f4b3aa6951679a0ed130a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d93b3d72b624006b7ac8293e0aa11e4"
I20260924 15:23:23.614717 9173 raft_consensus.cc:3037] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:23.615381 9173 raft_consensus.cc:2454] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd183621a15f4b3aa6951679a0ed130a in term 1.
I20260924 15:23:23.615491 9305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399eef3b77124dc28caaf7d60ae1608a" candidate_uuid: "cd183621a15f4b3aa6951679a0ed130a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4064f9851ec34f4ab8d9fe7788987f3c"
I20260924 15:23:23.615533 9370 leader_election.cc:302] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d93b3d72b624006b7ac8293e0aa11e4, cd183621a15f4b3aa6951679a0ed130a; no voters:
I20260924 15:23:23.615581 9305 raft_consensus.cc:3037] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:23.615711 9501 raft_consensus.cc:2781] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:23.615872 9501 raft_consensus.cc:686] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 1 LEADER]: Becoming Leader. State: Replica: cd183621a15f4b3aa6951679a0ed130a, State: Running, Role: LEADER
I20260924 15:23:23.615969 9501 consensus_queue.cc:237] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } }
I20260924 15:23:23.616540 9305 raft_consensus.cc:2454] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd183621a15f4b3aa6951679a0ed130a in term 1.
I20260924 15:23:23.616882 9031 catalog_manager.cc:5495] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a reported cstate change: term changed from 0 to 1, leader changed from <none> to cd183621a15f4b3aa6951679a0ed130a (127.4.78.67). New cstate: current_term: 1 leader_uuid: "cd183621a15f4b3aa6951679a0ed130a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd183621a15f4b3aa6951679a0ed130a" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40119 } health_report { overall_health: HEALTHY } } }
W20260924 15:23:23.633481 9220 tablet.cc:2369] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:23.666527 9495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.665589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57450 (local address 127.4.78.67:40119)
0924 15:23:23.665649 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.665650 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.665660 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:23.665698 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.665698 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.665711 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.665769 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.666328 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.666329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.666329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.666331 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.666343 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.666346 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.666385 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.666414 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.666424 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:23:23.668395 9305 raft_consensus.cc:1261] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 1 FOLLOWER]: Refusing update from remote peer cd183621a15f4b3aa6951679a0ed130a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:23.668396 9173 raft_consensus.cc:1261] T 399eef3b77124dc28caaf7d60ae1608a P 4d93b3d72b624006b7ac8293e0aa11e4 [term 1 FOLLOWER]: Refusing update from remote peer cd183621a15f4b3aa6951679a0ed130a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:23.668805 9506 consensus_queue.cc:1035] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d93b3d72b624006b7ac8293e0aa11e4" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 33953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:23.668944 9501 consensus_queue.cc:1035] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4064f9851ec34f4ab8d9fe7788987f3c" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43379 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:23.671433 9510 mvcc.cc:205] Tried to move back new op lower bound from 7332918901422817280 to 7332918901211680768. Current Snapshot: MvccSnapshot[applied={T|T < 7332918901422817280}]
I20260924 15:23:23.671201 9509 mvcc.cc:205] Tried to move back new op lower bound from 7332918901422817280 to 7332918901211680768. Current Snapshot: MvccSnapshot[applied={T|T < 7332918901422817280}]
I20260924 15:23:23.671546 9511 mvcc.cc:205] Tried to move back new op lower bound from 7332918901422817280 to 7332918901211680768. Current Snapshot: MvccSnapshot[applied={T|T < 7332918901422817280}]
I20260924 15:23:23.673593 9031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:23.673671 9031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:23.674443 9031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 944323237a804bc4b25ab1fd82562efb: Sending DeleteTablet for 3 replicas of tablet 399eef3b77124dc28caaf7d60ae1608a
I20260924 15:23:23.674759 9153 tablet_service.cc:1514] Processing DeleteTablet for tablet 399eef3b77124dc28caaf7d60ae1608a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:23 UTC) from {username='slave'} at 127.0.0.1:33924
I20260924 15:23:23.674923 9416 tablet_service.cc:1514] Processing DeleteTablet for tablet 399eef3b77124dc28caaf7d60ae1608a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:23 UTC) from {username='slave'} at 127.0.0.1:57436
I20260924 15:23:23.675036 9284 tablet_service.cc:1514] Processing DeleteTablet for tablet 399eef3b77124dc28caaf7d60ae1608a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:23 UTC) from {username='slave'} at 127.0.0.1:35752
I20260924 15:23:23.675100 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9091
I20260924 15:23:23.675822 9521 tablet_replica.cc:330] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: stopping tablet replica
I20260924 15:23:23.675930 9521 raft_consensus.cc:2229] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:23.676034 9521 raft_consensus.cc:2258] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:23.676440 9521 ts_tablet_manager.cc:1905] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:23.678382 9521 ts_tablet_manager.cc:1918] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:23.678463 9521 log.cc:1196] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/399eef3b77124dc28caaf7d60ae1608a
I20260924 15:23:23.678745 9521 ts_tablet_manager.cc:1939] T 399eef3b77124dc28caaf7d60ae1608a P 4064f9851ec34f4ab8d9fe7788987f3c: Deleting consensus metadata
I20260924 15:23:23.679287 9520 tablet_replica.cc:330] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: stopping tablet replica
I20260924 15:23:23.679309 9015 catalog_manager.cc:4841] TS 4064f9851ec34f4ab8d9fe7788987f3c (127.4.78.66:43379): tablet 399eef3b77124dc28caaf7d60ae1608a (table test-table [id=a3de630d1e974355b4f490903da399b3]) successfully deleted
I20260924 15:23:23.679385 9520 raft_consensus.cc:2229] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:23.679482 9520 raft_consensus.cc:2258] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:23.679943 9520 ts_tablet_manager.cc:1905] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:23.681748 9520 ts_tablet_manager.cc:1918] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:23.681893 9520 log.cc:1196] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/399eef3b77124dc28caaf7d60ae1608a
I20260924 15:23:23.682196 9520 ts_tablet_manager.cc:1939] T 399eef3b77124dc28caaf7d60ae1608a P cd183621a15f4b3aa6951679a0ed130a: Deleting consensus metadata
I20260924 15:23:23.682603 9018 catalog_manager.cc:4841] TS cd183621a15f4b3aa6951679a0ed130a (127.4.78.67:40119): tablet 399eef3b77124dc28caaf7d60ae1608a (table test-table [id=a3de630d1e974355b4f490903da399b3]) successfully deleted
W20260924 15:23:23.683203 9016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:33953 (error 108)
I20260924 15:23:23.683395 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9222
I20260924 15:23:23.683652 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.683525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:33953 (local address 127.0.0.1:33928)
0924 15:23:23.683570 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.683614 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:33953: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 15:23:23.683761 9016 catalog_manager.cc:4584] TS 4d93b3d72b624006b7ac8293e0aa11e4 (127.4.78.65:33953): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 399eef3b77124dc28caaf7d60ae1608a: Network error: Client connection negotiation failed: client connection to 127.4.78.65:33953: connect: Connection refused (error 111)
I20260924 15:23:23.688592 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9354
I20260924 15:23:23.693346 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9000
2026-09-24T15:23:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (827 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 15:23:23.712800 4409 mini_oidc.cc:152] Starting JWKS server
I20260924 15:23:23.713902 4409 webserver.cc:460] Webserver started at https://127.0.0.1:34303/ using document root <none> and password file <none>
I20260924 15:23:23.714002 4409 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:23:23.714139 4409 webserver.cc:460] Webserver started at http://127.0.0.1:33117/ using document root <none> and password file <none>
2026-09-24T15:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:23Z Disabled control of system clock
I20260924 15:23:23.721668 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:46471
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:44117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:46471
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34303/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/testchainca.cert with env {}
W20260924 15:23:23.800885 9528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:23.801082 9528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:23.801105 9528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:23.802253 9528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:23.802295 9528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:23.802307 9528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:23.802320 9528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:23.802332 9528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:23:23.802345 9528 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:23:23.802356 9528 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34303/jwks.json
I20260924 15:23:23.803785 9528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:46471
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:46471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34303/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:23.803969 9528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:23.804131 9528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:23.808717 9536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:23.808712 9534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:23.808712 9533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:23.808912 9528 server_base.cc:1034] running on GCE node
I20260924 15:23:23.809155 9528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:23.809468 9528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:23.810618 9528 hybrid_clock.cc:648] HybridClock initialized: now 1790263403810609 us; error 29 us; skew 500 ppm
I20260924 15:23:23.811740 9528 webserver.cc:460] Webserver started at http://127.4.78.126:39517/ using document root <none> and password file <none>
I20260924 15:23:23.811977 9528 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:23.812039 9528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:23.812178 9528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:23.813153 9528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "9e45fd2301eb4b4cac29b1f26feae5c0"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "be38a9bfeb8563bb483640fd81c03698"
server_key_iv: "9b207c95c9a64cb8f751fdb976dda41f"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.813511 9528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "9e45fd2301eb4b4cac29b1f26feae5c0"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "be38a9bfeb8563bb483640fd81c03698"
server_key_iv: "9b207c95c9a64cb8f751fdb976dda41f"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.814808 9528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:23.815478 9542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:23.815640 9528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:23.815732 9528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "9e45fd2301eb4b4cac29b1f26feae5c0"
format_stamp: "Formatted at 2026-09-24 15:23:23 on dist-test-slave-l766"
server_key: "be38a9bfeb8563bb483640fd81c03698"
server_key_iv: "9b207c95c9a64cb8f751fdb976dda41f"
server_key_version: "encryptionkey@0"
I20260924 15:23:23.815810 9528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:23.826922 9528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:23.827232 9528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:23.827394 9528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:23.831241 9528 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:46471
I20260924 15:23:23.831277 9594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:46471 every 8 connection(s)
I20260924 15:23:23.831595 9528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:23.832468 9595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:23.834695 9595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0: Bootstrap starting.
I20260924 15:23:23.835209 9595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:23.835464 9595 log.cc:824] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:23.835626 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9528
I20260924 15:23:23.835728 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:23.835937 4409 external_mini_cluster.cc:1428] Setting key 94128395c1af4991621c6ad7abea1cb2
I20260924 15:23:23.836264 9595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0: No bootstrap required, opened a new log
I20260924 15:23:23.837990 9595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45fd2301eb4b4cac29b1f26feae5c0" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 46471 } }
I20260924 15:23:23.838143 9595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:23.838197 9595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e45fd2301eb4b4cac29b1f26feae5c0, State: Initialized, Role: FOLLOWER
I20260924 15:23:23.838325 9595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45fd2301eb4b4cac29b1f26feae5c0" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 46471 } }
I20260924 15:23:23.838436 9595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:23.838480 9595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:23.838531 9595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:23.839030 9595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45fd2301eb4b4cac29b1f26feae5c0" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 46471 } }
I20260924 15:23:23.839140 9595 leader_election.cc:302] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e45fd2301eb4b4cac29b1f26feae5c0; no voters:
I20260924 15:23:23.839319 9595 leader_election.cc:288] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:23.839349 9600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:23.839491 9600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [term 1 LEADER]: Becoming Leader. State: Replica: 9e45fd2301eb4b4cac29b1f26feae5c0, State: Running, Role: LEADER
I20260924 15:23:23.839576 9595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:23.839680 9600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45fd2301eb4b4cac29b1f26feae5c0" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 46471 } }
I20260924 15:23:23.840392 9602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e45fd2301eb4b4cac29b1f26feae5c0. Latest consensus state: current_term: 1 leader_uuid: "9e45fd2301eb4b4cac29b1f26feae5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45fd2301eb4b4cac29b1f26feae5c0" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 46471 } } }
I20260924 15:23:23.840489 9602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:23.840651 9601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e45fd2301eb4b4cac29b1f26feae5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e45fd2301eb4b4cac29b1f26feae5c0" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 46471 } } }
I20260924 15:23:23.840710 9601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:23.840834 9609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:23.841478 9609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:23.842988 9609 catalog_manager.cc:1347] Generated new cluster ID: 0677623cfa284e64ba8066132ace7f9b
I20260924 15:23:23.843043 9609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:23.843348 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.836382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60194 (local address 127.4.78.126:46471)
0924 15:23:23.836572 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.836580 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.836604 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:23:23.837968 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.837974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.838035 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.838167 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:23.838372 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:23.840028 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:23.841183 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:23.841758 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:23.841938 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:23.842602 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:23.842606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:23.842606 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:23.842826 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:23.842867 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:23.842881 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:23.842982 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:23.843066 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:23:23.843143 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260924 15:23:23.844187 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:23:23.857071 9609 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:23.857549 9609 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:23.863224 9609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e45fd2301eb4b4cac29b1f26feae5c0: Generated new TSK 0
I20260924 15:23:23.863386 9609 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 15:23:23.883494 9625 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 15:23:23.883586 9625 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 15:23:23.885150 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:23.869051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.4.78.126:46471)
0924 15:23:23.869094 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:23.869096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:23.869233 (+ 137us) server_negotiation.cc:407] Connection header received
0924 15:23:23.869330 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:23.869330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:23.869340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:23.869391 (+ 51us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:23:23.869520 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:23.870038 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:23.871184 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:23.871356 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:23.871403 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:23.871596 (+ 193us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:23:23.884979 (+ 13383us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 15:23:23.885095 (+ 116us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60208: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 15:23:23.885215 9599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60208: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 15:23:23.885515 9624 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:23.868955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:46471 (local address 127.0.0.1:60208)
0924 15:23:23.869163 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:23.869175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:23.869243 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:23.869410 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:23.869411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:23.869413 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:23:23.869469 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:23.869471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:23.870105 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:23.870107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:23.871109 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:23.871112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:23.871422 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:23.871423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:23.871483 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:23.871504 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:23:23.885089 (+ 13585us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 15:23:23.885206 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.78.126:46471: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":98,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.78.126:46471: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34303/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 15:23:23.885965 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9528
2026-09-24T15:23:23Z chronyd exiting
I20260924 15:23:23.922088 4409 test_util.cc:175] -----------------------------------------------
I20260924 15:23:23.922173 4409 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263382361574-4409-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 15:23:23.922948 4409 mini_oidc.cc:152] Starting JWKS server
I20260924 15:23:23.923632 4409 webserver.cc:460] Webserver started at https://127.0.0.1:42273/ using document root <none> and password file <none>
I20260924 15:23:23.923681 4409 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:23:23.923871 4409 webserver.cc:460] Webserver started at http://127.0.0.1:41157/ using document root <none> and password file <none>
2026-09-24T15:23:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:23Z Disabled control of system clock
I20260924 15:23:23.931334 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:33041
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:34529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:33041
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42273/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/testchainca.cert with env {}
W20260924 15:23:24.009366 9631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:24.009548 9631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:24.009570 9631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:24.010659 9631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:24.010695 9631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:24.010707 9631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:24.010720 9631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:24.010731 9631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:23:24.010743 9631 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:23:24.010756 9631 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42273/jwks.json
I20260924 15:23:24.012140 9631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:33041
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:33041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42273/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:24.012326 9631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:24.012477 9631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:24.017022 9639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:24.017115 9636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:24.017021 9637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:24.017268 9631 server_base.cc:1034] running on GCE node
I20260924 15:23:24.017561 9631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:24.017886 9631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:24.019045 9631 hybrid_clock.cc:648] HybridClock initialized: now 1790263404019031 us; error 42 us; skew 500 ppm
I20260924 15:23:24.020208 9631 webserver.cc:460] Webserver started at http://127.4.78.126:33037/ using document root <none> and password file <none>
I20260924 15:23:24.020439 9631 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:24.020503 9631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:24.020632 9631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:24.021627 9631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "405404b222e6440c81653dbbdcc35054"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "003aad780a7f7cc73a8902381bb7663f"
server_key_iv: "e76cdc90d39378ffe7b16de0c9ced6ac"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.021996 9631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "405404b222e6440c81653dbbdcc35054"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "003aad780a7f7cc73a8902381bb7663f"
server_key_iv: "e76cdc90d39378ffe7b16de0c9ced6ac"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.023244 9631 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:24.023954 9645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:24.024127 9631 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:24.024224 9631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "405404b222e6440c81653dbbdcc35054"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "003aad780a7f7cc73a8902381bb7663f"
server_key_iv: "e76cdc90d39378ffe7b16de0c9ced6ac"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.024305 9631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:24.042376 9631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:24.042712 9631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:24.042874 9631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:24.046717 9631 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:33041
I20260924 15:23:24.046754 9697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:33041 every 8 connection(s)
I20260924 15:23:24.047106 9631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:24.048038 9698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:24.051105 9698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054: Bootstrap starting.
I20260924 15:23:24.051777 9698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:24.052116 9698 log.cc:824] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:24.052824 9698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054: No bootstrap required, opened a new log
I20260924 15:23:24.054497 9698 raft_consensus.cc:348] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405404b222e6440c81653dbbdcc35054" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33041 } }
I20260924 15:23:24.054656 9698 raft_consensus.cc:374] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:24.054701 9698 raft_consensus.cc:729] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 405404b222e6440c81653dbbdcc35054, State: Initialized, Role: FOLLOWER
I20260924 15:23:24.054847 9698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405404b222e6440c81653dbbdcc35054" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33041 } }
I20260924 15:23:24.054948 9698 raft_consensus.cc:388] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:24.054992 9698 raft_consensus.cc:482] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:24.055048 9698 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:24.055179 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9631
I20260924 15:23:24.055279 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:24.055444 4409 external_mini_cluster.cc:1428] Setting key 2a108752205556ed10a32812319d4c15
I20260924 15:23:24.055742 9698 raft_consensus.cc:504] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405404b222e6440c81653dbbdcc35054" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33041 } }
I20260924 15:23:24.055858 9698 leader_election.cc:302] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 405404b222e6440c81653dbbdcc35054; no voters:
I20260924 15:23:24.056015 9698 leader_election.cc:288] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:24.056046 9702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:24.056207 9702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [term 1 LEADER]: Becoming Leader. State: Replica: 405404b222e6440c81653dbbdcc35054, State: Running, Role: LEADER
I20260924 15:23:24.056291 9698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:24.056290 9702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405404b222e6440c81653dbbdcc35054" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33041 } }
I20260924 15:23:24.056636 9705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "405404b222e6440c81653dbbdcc35054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405404b222e6440c81653dbbdcc35054" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33041 } } }
I20260924 15:23:24.056968 9705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:24.056895 9704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 405404b222e6440c81653dbbdcc35054. Latest consensus state: current_term: 1 leader_uuid: "405404b222e6440c81653dbbdcc35054" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "405404b222e6440c81653dbbdcc35054" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33041 } } }
I20260924 15:23:24.057227 9704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:24.057394 9712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:24.057910 9712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:24.059429 9712 catalog_manager.cc:1347] Generated new cluster ID: 4a69972689294ec5a8432466820831c4
I20260924 15:23:24.059479 9712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:24.061755 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:24.055894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.4.78.126:33041)
0924 15:23:24.056130 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:24.056139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:24.056175 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:23:24.056301 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:24.056308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:24.056404 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:24.056780 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:24.056891 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.058354 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.059407 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.060062 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.060515 (+ 453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.060910 (+ 395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:24.060914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:24.060914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:24.061117 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:24.061159 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:24.061174 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:24.061428 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:24.061462 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 15:23:24.061541 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
I20260924 15:23:24.063985 9712 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:24.064461 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:23:24.064616 9712 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:24.068975 9712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 405404b222e6440c81653dbbdcc35054: Generated new TSK 0
I20260924 15:23:24.069137 9712 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 15:23:24.097872 9728 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 15:23:24.097958 9728 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 15:23:24.098279 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:24.075643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.4.78.126:33041)
0924 15:23:24.075706 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:24.075708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:24.075826 (+ 118us) server_negotiation.cc:407] Connection header received
0924 15:23:24.075900 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:24.075901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:24.075911 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:24.075968 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:23:24.076105 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.076582 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.077697 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.077942 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.077992 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.078183 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:23:24.098060 (+ 19877us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 15:23:24.098227 (+ 167us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38994: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 15:23:24.098345 9703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38994: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 15:23:24.098410 9727 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:24.075589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:33041 (local address 127.0.0.1:38994)
0924 15:23:24.075756 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:24.075768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:24.075834 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:24.075981 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:24.075982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:24.075984 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:23:24.076044 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:24.076046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.076660 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.076661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:24.077619 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:24.077622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.078008 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.078009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:24.078078 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.078084 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:23:24.098182 (+ 20098us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 15:23:24.098309 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.78.126:33041: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42273/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260924 15:23:24.098716 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9631
2026-09-24T15:23:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (213 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 15:23:24.136174 4409 mini_oidc.cc:152] Starting JWKS server
I20260924 15:23:24.136865 4409 webserver.cc:460] Webserver started at https://127.0.0.1:42075/ using document root <none> and password file <none>
I20260924 15:23:24.136906 4409 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:23:24.137085 4409 webserver.cc:460] Webserver started at http://127.0.0.1:34037/ using document root <none> and password file <none>
2026-09-24T15:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:24Z Disabled control of system clock
I20260924 15:23:24.144388 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42375
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:37255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:42375
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42075/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 15:23:24.223060 9734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:24.223300 9734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:24.223337 9734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:24.224495 9734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:24.224567 9734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:24.224591 9734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:24.224620 9734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:24.224649 9734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:23:24.224676 9734 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:23:24.224704 9734 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42075/jwks.json
I20260924 15:23:24.226099 9734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:37255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:42375
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42075/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:24.226365 9734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:24.226635 9734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:24.231407 9742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:24.231423 9740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:24.231423 9739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:24.231755 9734 server_base.cc:1034] running on GCE node
I20260924 15:23:24.232014 9734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:24.232300 9734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:24.233475 9734 hybrid_clock.cc:648] HybridClock initialized: now 1790263404233462 us; error 35 us; skew 500 ppm
I20260924 15:23:24.234684 9734 webserver.cc:460] Webserver started at http://127.4.78.126:46409/ using document root <none> and password file <none>
I20260924 15:23:24.234920 9734 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:24.234985 9734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:24.235123 9734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:24.236147 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "75b329ba3d734788bf80234f39f0adfc"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "f70334be5c7149b4ee21031c5ba985ad"
server_key_iv: "922b0a4e9bdd85e4a8a10319ec2c660a"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.236511 9734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "75b329ba3d734788bf80234f39f0adfc"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "f70334be5c7149b4ee21031c5ba985ad"
server_key_iv: "922b0a4e9bdd85e4a8a10319ec2c660a"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.237751 9734 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:24.238530 9748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:24.238695 9734 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:24.238791 9734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "75b329ba3d734788bf80234f39f0adfc"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "f70334be5c7149b4ee21031c5ba985ad"
server_key_iv: "922b0a4e9bdd85e4a8a10319ec2c660a"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.238878 9734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:24.252987 9734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:24.253275 9734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:24.253446 9734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:24.257270 9734 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:42375
I20260924 15:23:24.257294 9800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:42375 every 8 connection(s)
I20260924 15:23:24.257603 9734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:24.258307 9801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:24.258677 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9734
I20260924 15:23:24.258776 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:24.258944 4409 external_mini_cluster.cc:1428] Setting key dd291e94765b639ec40b29367183af87
I20260924 15:23:24.261977 9801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc: Bootstrap starting.
I20260924 15:23:24.262571 9801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:24.262873 9801 log.cc:824] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:24.263545 9801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc: No bootstrap required, opened a new log
I20260924 15:23:24.264701 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:24.259399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56004 (local address 127.4.78.126:42375)
0924 15:23:24.259618 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:24.259626 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:24.259658 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:23:24.260186 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:24.260195 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:24.260283 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:24.260425 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:24.260588 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.261940 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.262775 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.263322 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.263439 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.264026 (+ 587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:24.264030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:24.264030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:24.264248 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:24.264289 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:24.264302 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:24.264363 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:24.264425 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:23:24.264501 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":91,"threads_started":1}
I20260924 15:23:24.265758 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:23:24.266023 9801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b329ba3d734788bf80234f39f0adfc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42375 } }
I20260924 15:23:24.266173 9801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:24.266208 9801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b329ba3d734788bf80234f39f0adfc, State: Initialized, Role: FOLLOWER
I20260924 15:23:24.266333 9801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b329ba3d734788bf80234f39f0adfc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42375 } }
I20260924 15:23:24.266403 9801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:24.266433 9801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:24.266479 9801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:24.267042 9801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b329ba3d734788bf80234f39f0adfc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42375 } }
I20260924 15:23:24.267145 9801 leader_election.cc:302] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75b329ba3d734788bf80234f39f0adfc; no voters:
I20260924 15:23:24.267284 9801 leader_election.cc:288] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:24.267324 9806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:24.267488 9806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [term 1 LEADER]: Becoming Leader. State: Replica: 75b329ba3d734788bf80234f39f0adfc, State: Running, Role: LEADER
I20260924 15:23:24.267557 9801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:24.267570 9806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b329ba3d734788bf80234f39f0adfc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42375 } }
I20260924 15:23:24.267992 9807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75b329ba3d734788bf80234f39f0adfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b329ba3d734788bf80234f39f0adfc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42375 } } }
I20260924 15:23:24.268090 9807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:24.268270 9808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75b329ba3d734788bf80234f39f0adfc. Latest consensus state: current_term: 1 leader_uuid: "75b329ba3d734788bf80234f39f0adfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75b329ba3d734788bf80234f39f0adfc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42375 } } }
I20260924 15:23:24.268340 9808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:24.268695 9813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:24.269198 9813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:24.270772 9813 catalog_manager.cc:1347] Generated new cluster ID: 1247d40a37164d47bb0c9022ef541401
I20260924 15:23:24.270828 9813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:24.295234 9813 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:24.295782 9813 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:24.301221 9813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75b329ba3d734788bf80234f39f0adfc: Generated new TSK 0
I20260924 15:23:24.301409 9813 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:23:24.313611 9804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:24.310378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56008 (local address 127.4.78.126:42375)
0924 15:23:24.310438 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:24.310440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:24.310547 (+ 107us) server_negotiation.cc:407] Connection header received
0924 15:23:24.310628 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:24.310629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:24.310646 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:24.310715 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:23:24.310873 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.311360 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.312164 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.312369 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.312439 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.312622 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:23:24.313392 (+ 770us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42075/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 15:23:24.313541 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42075/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 15:23:24.313678 9804 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56008: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42075/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 15:23:24.313979 9830 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:24.310322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42375 (local address 127.0.0.1:56008)
0924 15:23:24.310486 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:24.310495 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:24.310565 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:24.310728 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:24.310729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:24.310731 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:23:24.310806 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:24.310809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.311459 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.311460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:24.312079 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:24.312081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.312458 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.312459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:24.312521 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.312526 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:23:24.313493 (+ 967us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42075/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 15:23:24.313584 (+ 91us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.78.126:42375: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42075/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260924 15:23:24.314220 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9734
E20260924 15:23:24.326434 9831 webserver.cc:533] Webserver: sslize failed:
E20260924 15:23:24.326498 9831 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T15:23:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (215 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T15:23:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:24Z Disabled control of system clock
I20260924 15:23:24.359669 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:44565
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:44565
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/testchainca.cert with env {}
W20260924 15:23:24.439043 9835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:24.439239 9835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:24.439261 9835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:24.440416 9835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:24.440459 9835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:24.440472 9835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:24.440490 9835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:24.440546 9835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:23:24.440578 9835 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:23:24.440616 9835 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 15:23:24.442261 9835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:44565
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:44565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:24.442508 9835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:24.442713 9835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:24.447311 9840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:24.447365 9841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:24.447418 9835 server_base.cc:1034] running on GCE node
W20260924 15:23:24.447365 9843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:24.447741 9835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:24.448009 9835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:24.449159 9835 hybrid_clock.cc:648] HybridClock initialized: now 1790263404449139 us; error 38 us; skew 500 ppm
I20260924 15:23:24.450354 9835 webserver.cc:460] Webserver started at http://127.4.78.126:44301/ using document root <none> and password file <none>
I20260924 15:23:24.450536 9835 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:24.450575 9835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:24.450656 9835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:24.451623 9835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "b95c8a92986540aeb1012636724f3335"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "4defb96d494a35976d9b29f7f4588956"
server_key_iv: "f3bfe8c6b892440602f8740968a5d2af"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.451925 9835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "b95c8a92986540aeb1012636724f3335"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "4defb96d494a35976d9b29f7f4588956"
server_key_iv: "f3bfe8c6b892440602f8740968a5d2af"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.453100 9835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:23:24.453729 9849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:24.453943 9835 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:24.454015 9835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "b95c8a92986540aeb1012636724f3335"
format_stamp: "Formatted at 2026-09-24 15:23:24 on dist-test-slave-l766"
server_key: "4defb96d494a35976d9b29f7f4588956"
server_key_iv: "f3bfe8c6b892440602f8740968a5d2af"
server_key_version: "encryptionkey@0"
I20260924 15:23:24.454066 9835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:24.474187 9835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:24.474431 9835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:24.474542 9835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:24.478372 9835 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:44565
I20260924 15:23:24.478402 9902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:44565 every 8 connection(s)
I20260924 15:23:24.478741 9835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:24.479636 9903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:24.481726 9903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335: Bootstrap starting.
I20260924 15:23:24.482307 9903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:24.482535 9903 log.cc:824] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:24.483139 9903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335: No bootstrap required, opened a new log
I20260924 15:23:24.483934 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9835
I20260924 15:23:24.484059 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:24.484248 4409 external_mini_cluster.cc:1428] Setting key 67c5934763601fbd47b103ddde72a37c
I20260924 15:23:24.485101 9903 raft_consensus.cc:348] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b95c8a92986540aeb1012636724f3335" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44565 } }
I20260924 15:23:24.485260 9903 raft_consensus.cc:374] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:24.485299 9903 raft_consensus.cc:729] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95c8a92986540aeb1012636724f3335, State: Initialized, Role: FOLLOWER
I20260924 15:23:24.485430 9903 consensus_queue.cc:260] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b95c8a92986540aeb1012636724f3335" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44565 } }
I20260924 15:23:24.485512 9903 raft_consensus.cc:388] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:24.485561 9903 raft_consensus.cc:482] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:24.485610 9903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:24.486176 9903 raft_consensus.cc:504] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b95c8a92986540aeb1012636724f3335" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44565 } }
I20260924 15:23:24.486289 9903 leader_election.cc:302] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b95c8a92986540aeb1012636724f3335; no voters:
I20260924 15:23:24.486433 9903 leader_election.cc:288] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:24.486480 9908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:24.486624 9908 raft_consensus.cc:686] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [term 1 LEADER]: Becoming Leader. State: Replica: b95c8a92986540aeb1012636724f3335, State: Running, Role: LEADER
I20260924 15:23:24.486723 9903 sys_catalog.cc:564] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:24.486707 9908 consensus_queue.cc:237] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b95c8a92986540aeb1012636724f3335" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44565 } }
I20260924 15:23:24.487066 9910 sys_catalog.cc:455] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b95c8a92986540aeb1012636724f3335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b95c8a92986540aeb1012636724f3335" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44565 } } }
I20260924 15:23:24.487164 9910 sys_catalog.cc:458] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:24.487054 9908 sys_catalog.cc:455] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b95c8a92986540aeb1012636724f3335. Latest consensus state: current_term: 1 leader_uuid: "b95c8a92986540aeb1012636724f3335" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b95c8a92986540aeb1012636724f3335" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44565 } } }
I20260924 15:23:24.487366 9908 sys_catalog.cc:458] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:24.487476 9914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:24.488011 9914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:24.489617 9914 catalog_manager.cc:1347] Generated new cluster ID: a1c84eb2374d43c9b9e0bbd1d55f1c2c
I20260924 15:23:24.489663 9914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:24.491443 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:24.484748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.4.78.126:44565)
0924 15:23:24.484975 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:24.484984 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:24.485019 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:23:24.485130 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:24.485137 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:24.485215 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:24.485357 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:24.485676 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.488341 (+ 2665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.489231 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.489943 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.490061 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.490759 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:24.490762 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:24.490763 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:24.490976 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:24.491013 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:24.491026 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:24.491086 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:24.491182 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:23:24.491252 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260924 15:23:24.492343 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:23:24.512905 9914 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:24.513620 9914 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:24.520339 9914 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b95c8a92986540aeb1012636724f3335: Generated new TSK 0
I20260924 15:23:24.520540 9914 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:23:24.622126 9907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:24.530040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41840 (local address 127.4.78.126:44565)
0924 15:23:24.530104 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:24.530106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:24.530201 (+ 95us) server_negotiation.cc:407] Connection header received
0924 15:23:24.530288 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:24.530289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:24.530302 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:24.530385 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:23:24.530525 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.531015 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.531802 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.532011 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.532084 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.532257 (+ 173us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:23:24.621841 (+ 89584us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 15:23:24.622039 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41840: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 15:23:24.622213 9907 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41840: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 15:23:24.622226 9932 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:24.529969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:44565 (local address 127.0.0.1:41840)
0924 15:23:24.530136 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:24.530145 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:24.530220 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:24.530384 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:24.530385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:24.530387 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:23:24.530455 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:24.530457 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.531097 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.531098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:24.531696 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:24.531698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:24.532089 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:24.532090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:24.532152 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:24.532157 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:23:24.621985 (+ 89828us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 15:23:24.622108 (+ 123us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.78.126:44565: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260924 15:23:24.622512 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9835
2026-09-24T15:23:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (287 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:24 dist-test-slave-l766 krb5kdc[9945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:24 dist-test-slave-l766 krb5kdc[9945](info): set up 2 sockets
Sep 24 15:23:24 dist-test-slave-l766 krb5kdc[9945](info): commencing operation
krb5kdc: starting...
W20260924 15:23:25.824990 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:25 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:25.936868 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:32969
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:32969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:26.015614 9958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:26.015844 9958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:26.015882 9958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:26.017045 9958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:26.017110 9958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:26.017134 9958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:26.017164 9958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:26.017191 9958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:26.018849 9958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:32969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:32969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:26.019122 9958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:26.019342 9958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:26.024091 9966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:26.024084 9963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:26.024168 9964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:26.024256 9958 server_base.cc:1034] running on GCE node
I20260924 15:23:26.024528 9958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:26.024802 9958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:26.025991 9958 hybrid_clock.cc:648] HybridClock initialized: now 1790263406025972 us; error 27 us; skew 500 ppm
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.035336 9958 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:26.035784 9958 webserver.cc:460] Webserver started at http://127.4.78.126:44803/ using document root <none> and password file <none>
I20260924 15:23:26.036022 9958 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:26.036093 9958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:26.036228 9958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:26.037276 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e262ad4b4629407f86c6ef7163e1bc06"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "4c47f5276dacf267199ecac501283752"
server_key_iv: "d8a475f8936037863543b5bf2f4d0eaf"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.037650 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e262ad4b4629407f86c6ef7163e1bc06"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "4c47f5276dacf267199ecac501283752"
server_key_iv: "d8a475f8936037863543b5bf2f4d0eaf"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.039000 9958 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:26.039701 9974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.039878 9958 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.039975 9958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e262ad4b4629407f86c6ef7163e1bc06"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "4c47f5276dacf267199ecac501283752"
server_key_iv: "d8a475f8936037863543b5bf2f4d0eaf"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.040058 9958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:26.048341 9958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:26.050688 9958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:26.050849 9958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:26.054801 9958 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:32969
I20260924 15:23:26.054848 10026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:32969 every 8 connection(s)
I20260924 15:23:26.055208 9958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:26.056088 10027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:26.059510 10027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06: Bootstrap starting.
I20260924 15:23:26.060020 10027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:26.060258 10027 log.cc:824] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:26.060837 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 9958
I20260924 15:23:26.060914 10027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06: No bootstrap required, opened a new log
I20260924 15:23:26.060952 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:26.061134 4409 external_mini_cluster.cc:1428] Setting key 666ddf0d4786d84d33b4e0ef2b021d78
I20260924 15:23:26.063287 10027 raft_consensus.cc:348] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e262ad4b4629407f86c6ef7163e1bc06" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 32969 } }
I20260924 15:23:26.063472 10027 raft_consensus.cc:374] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:26.063519 10027 raft_consensus.cc:729] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e262ad4b4629407f86c6ef7163e1bc06, State: Initialized, Role: FOLLOWER
I20260924 15:23:26.063668 10027 consensus_queue.cc:260] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e262ad4b4629407f86c6ef7163e1bc06" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 32969 } }
I20260924 15:23:26.063759 10027 raft_consensus.cc:388] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:26.063793 10027 raft_consensus.cc:482] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:26.063850 10027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:26.064884 10027 raft_consensus.cc:504] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e262ad4b4629407f86c6ef7163e1bc06" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 32969 } }
I20260924 15:23:26.065013 10027 leader_election.cc:302] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e262ad4b4629407f86c6ef7163e1bc06; no voters:
I20260924 15:23:26.065214 10027 leader_election.cc:288] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:26.065249 10033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:26.065362 10033 raft_consensus.cc:686] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [term 1 LEADER]: Becoming Leader. State: Replica: e262ad4b4629407f86c6ef7163e1bc06, State: Running, Role: LEADER
I20260924 15:23:26.065451 10033 consensus_queue.cc:237] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e262ad4b4629407f86c6ef7163e1bc06" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 32969 } }
I20260924 15:23:26.065501 10027 sys_catalog.cc:564] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:26.065801 10035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e262ad4b4629407f86c6ef7163e1bc06. Latest consensus state: current_term: 1 leader_uuid: "e262ad4b4629407f86c6ef7163e1bc06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e262ad4b4629407f86c6ef7163e1bc06" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 32969 } } }
I20260924 15:23:26.066069 10035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:26.066002 10034 sys_catalog.cc:455] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e262ad4b4629407f86c6ef7163e1bc06" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e262ad4b4629407f86c6ef7163e1bc06" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 32969 } } }
I20260924 15:23:26.066304 10034 sys_catalog.cc:458] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:26.066334 10041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:26.066880 10041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:26.069036 10041 catalog_manager.cc:1347] Generated new cluster ID: eaedc29c9b9c48c594049d61635d4735
I20260924 15:23:26.069090 10041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263405, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:26.073505 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.061679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58940 (local address 127.4.78.126:32969)
0924 15:23:26.061899 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.061909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.062571 (+ 662us) server_negotiation.cc:407] Connection header received
0924 15:23:26.063253 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.063260 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.063306 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.063421 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.064293 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.065311 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.066478 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.067093 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.067372 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.070035 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:26.070039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:26.070040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:26.070051 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:26.071739 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.072140 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.072149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.072150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.072245 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.072459 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.072460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.072461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.072708 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:26.072883 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:26.072963 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:23:26.073332 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":135,"threads_started":1}
I20260924 15:23:26.085915 10041 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:26.086442 10041 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:26.088646 10041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e262ad4b4629407f86c6ef7163e1bc06: Generated new TSK 0
I20260924 15:23:26.088831 10041 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:26.113803 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:32969
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:26.193261 10054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:26.193447 10054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:26.193470 10054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:26.194614 10054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:26.194653 10054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:26.194679 10054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:26.196096 10054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:26.196348 10054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:26.196566 10054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:26.201409 10062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:26.201381 10059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:26.201381 10060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:26.201949 10054 server_base.cc:1034] running on GCE node
I20260924 15:23:26.202189 10054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:26.202450 10054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:26.203584 10054 hybrid_clock.cc:648] HybridClock initialized: now 1790263406203567 us; error 38 us; skew 500 ppm
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.206202 10054 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:26.206633 10054 webserver.cc:460] Webserver started at http://127.4.78.65:37089/ using document root <none> and password file <none>
I20260924 15:23:26.206809 10054 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:26.206849 10054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:26.206933 10054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:26.207899 10054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "ac7ab519531e4f858ff357557c242a41"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "f24095318beee1e54772288f39f55892"
server_key_iv: "8a54d72e1ba6fc00529bfb7f0b171423"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.208262 10054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "ac7ab519531e4f858ff357557c242a41"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "f24095318beee1e54772288f39f55892"
server_key_iv: "8a54d72e1ba6fc00529bfb7f0b171423"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.209602 10054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 15:23:26.210402 10069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.210565 10054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.210662 10054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "ac7ab519531e4f858ff357557c242a41"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "f24095318beee1e54772288f39f55892"
server_key_iv: "8a54d72e1ba6fc00529bfb7f0b171423"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.210739 10054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:26.228660 10054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:26.230953 10054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:26.231112 10054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:26.231340 10054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:26.231642 10054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:26.231699 10054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.231740 10054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:26.231768 10054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.237637 10054 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:40793
I20260924 15:23:26.237692 10184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:40793 every 8 connection(s)
I20260924 15:23:26.237998 10054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:26.238517 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 10054
I20260924 15:23:26.238622 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:26.238839 4409 external_mini_cluster.cc:1428] Setting key d86abf1ba1c4cbcf6d5802a513df72b8
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:26.248471 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.238830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:39871 (local address 127.4.78.126:32969)
0924 15:23:26.238903 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.238905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.239217 (+ 312us) server_negotiation.cc:407] Connection header received
0924 15:23:26.240262 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.240263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.240280 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.240348 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.241125 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.241594 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.242704 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.242969 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.243040 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.245342 (+ 2302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:26.245343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:26.245344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:26.245352 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:26.247119 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.247502 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.247504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.247504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.247571 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.247801 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.247802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.247802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.247944 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:26.248021 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:26.248262 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 15:23:26.248347 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:23:26.248481 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.238656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:32969 (local address 127.4.78.65:39871)
0924 15:23:26.239080 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:26.239131 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:26.240153 (+ 1022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:26.240406 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:26.240409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:26.240742 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:26.241030 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:26.241036 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.241686 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.241687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:26.242591 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:26.242593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.243054 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.243055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:26.243196 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.243673 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:26.243683 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:26.245246 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:26.247213 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:26.247218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:26.247220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:26.247427 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:26.247642 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:26.247642 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:26.247643 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:26.247736 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:26.248012 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:26.248016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:26.248127 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 15:23:26.248208 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:23:26.248324 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":82,"threads_started":1}
I20260924 15:23:26.249066 10185 heartbeater.cc:347] Connected to a master server at 127.4.78.126:32969
I20260924 15:23:26.249176 10185 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
I20260924 15:23:26.249377 10185 heartbeater.cc:511] Master 127.4.78.126:32969 requested a full tablet report, sending...
I20260924 15:23:26.249943 9991 ts_manager.cc:194] Registered new tserver with Master: ac7ab519531e4f858ff357557c242a41 (127.4.78.65:40793)
I20260924 15:23:26.250577 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:39871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:26.267055 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:32969
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:26.346189 10190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:26.346376 10190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:26.346400 10190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:26.347474 10190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:26.347506 10190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:26.347532 10190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:26.348883 10190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:26.349063 10190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:26.349223 10190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:26.354012 10196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:26.354020 10198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:26.354154 10190 server_base.cc:1034] running on GCE node
W20260924 15:23:26.354072 10195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:26.354498 10190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:26.354745 10190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:26.355918 10190 hybrid_clock.cc:648] HybridClock initialized: now 1790263406355904 us; error 26 us; skew 500 ppm
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.358408 10190 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:26.358860 10190 webserver.cc:460] Webserver started at http://127.4.78.66:34001/ using document root <none> and password file <none>
I20260924 15:23:26.359073 10190 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:26.359138 10190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:26.359295 10190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:26.360287 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "eda246c8a94a428fb3dee2d69d8cce83"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "f086d88df4a5f50ab107df528c066db1"
server_key_iv: "933ea10f9cab8f2399ccf97ea9cf6b72"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.360642 10190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "eda246c8a94a428fb3dee2d69d8cce83"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "f086d88df4a5f50ab107df528c066db1"
server_key_iv: "933ea10f9cab8f2399ccf97ea9cf6b72"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.361932 10190 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:23:26.362644 10205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.362811 10190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:26.362906 10190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "eda246c8a94a428fb3dee2d69d8cce83"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "f086d88df4a5f50ab107df528c066db1"
server_key_iv: "933ea10f9cab8f2399ccf97ea9cf6b72"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.362991 10190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:26.382733 10190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:26.384541 10190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:26.384701 10190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:26.384935 10190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:26.385243 10190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:26.385295 10190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.385335 10190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:26.385365 10190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.390954 10190 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:32817
I20260924 15:23:26.390995 10320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:32817 every 8 connection(s)
I20260924 15:23:26.391299 10190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:26.391518 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 10190
I20260924 15:23:26.391623 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:26.391841 4409 external_mini_cluster.cc:1428] Setting key daacf2a7de8fdf209b2df578a62c479b
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
I20260924 15:23:26.401747 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.392092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:51883 (local address 127.4.78.126:32969)
0924 15:23:26.392168 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.392171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.392378 (+ 207us) server_negotiation.cc:407] Connection header received
0924 15:23:26.393363 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.393364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.393378 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.393447 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.394236 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.394723 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.396267 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.396451 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.396505 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.398892 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:26.398894 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:26.398894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:26.398902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:26.400316 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.400719 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.400720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.400721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.400782 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.401035 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.401036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.401036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.401233 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:26.401296 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:26.401569 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 15:23:26.401656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:23:26.402012 10323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.392064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:32969 (local address 127.4.78.66:51883)
0924 15:23:26.392233 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:26.392283 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:26.393251 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:26.393505 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:26.393508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:26.393846 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:26.394143 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:26.394150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.394816 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.394817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:26.396185 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:26.396189 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.396535 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.396537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:26.396687 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.397161 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:26.397172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:26.398795 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:26.400402 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:26.400409 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:26.400411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:26.400623 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:26.400856 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:26.400857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:26.400858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:26.400961 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:26.401297 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:26.401304 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:26.401440 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 15:23:26.401547 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 15:23:26.401707 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260924 15:23:26.402504 10321 heartbeater.cc:347] Connected to a master server at 127.4.78.126:32969
I20260924 15:23:26.402603 10321 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:26.402805 10321 heartbeater.cc:511] Master 127.4.78.126:32969 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
I20260924 15:23:26.403196 9991 ts_manager.cc:194] Registered new tserver with Master: eda246c8a94a428fb3dee2d69d8cce83 (127.4.78.66:32817)
I20260924 15:23:26.403581 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:51883
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:26.417088 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:32969
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:26.497365 10326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:26.497560 10326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:26.497583 10326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:26.498822 10326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:26.498864 10326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:26.498890 10326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:26.500383 10326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:32969
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:26.500617 10326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:26.500833 10326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:26.505893 10332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:26.505950 10331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:26.505894 10334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:26.506110 10326 server_base.cc:1034] running on GCE node
I20260924 15:23:26.506330 10326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:26.506579 10326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:26.507730 10326 hybrid_clock.cc:648] HybridClock initialized: now 1790263406507705 us; error 39 us; skew 500 ppm
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.510519 10326 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:26.511022 10326 webserver.cc:460] Webserver started at http://127.4.78.67:46479/ using document root <none> and password file <none>
I20260924 15:23:26.511250 10326 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:26.511322 10326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:26.511459 10326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:26.512504 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "dd72f76b5e4f4768a5058696fdc291fd"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "328514e17579de72fd5af00636be42de"
server_key_iv: "244577b83cb7fd89de76260ceed296c0"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.512871 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "dd72f76b5e4f4768a5058696fdc291fd"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "328514e17579de72fd5af00636be42de"
server_key_iv: "244577b83cb7fd89de76260ceed296c0"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.514385 10326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:26.515133 10341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.515302 10326 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:26.515401 10326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "dd72f76b5e4f4768a5058696fdc291fd"
format_stamp: "Formatted at 2026-09-24 15:23:26 on dist-test-slave-l766"
server_key: "328514e17579de72fd5af00636be42de"
server_key_iv: "244577b83cb7fd89de76260ceed296c0"
server_key_version: "encryptionkey@0"
I20260924 15:23:26.515483 10326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:26.535372 10326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:26.537298 10326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:26.537462 10326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:26.537708 10326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:26.538048 10326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:26.538105 10326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.538146 10326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:26.538174 10326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:26.544269 10326 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:46381
I20260924 15:23:26.544304 10456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:46381 every 8 connection(s)
I20260924 15:23:26.544621 10326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:26.551388 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 10326
I20260924 15:23:26.551484 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:26.551733 4409 external_mini_cluster.cc:1428] Setting key 18af3ecb5f53f458d770da2c1c9468f4
I20260924 15:23:26.553996 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.545332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:54401 (local address 127.4.78.126:32969)
0924 15:23:26.545392 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.545393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.545633 (+ 240us) server_negotiation.cc:407] Connection header received
0924 15:23:26.546718 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.546719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.546731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.546785 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.547401 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.547904 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.548897 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.549116 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.549198 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.551180 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:26.551182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:26.551182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:26.551205 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:26.552861 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.553207 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.553208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.553208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.553257 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.553457 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.553457 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.553457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.553549 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:26.553597 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:26.553862 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 15:23:26.553936 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:23:26.554237 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.545307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:32969 (local address 127.4.78.67:54401)
0924 15:23:26.545483 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:26.545548 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:26.546600 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:26.546855 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:26.546859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:26.547121 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:26.547306 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:26.547311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.547999 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.548000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:26.548803 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:26.548805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.549203 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.549204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:26.549334 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.549767 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:26.549775 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:26.551101 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:26.552927 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:26.552933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:26.552934 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:26.553146 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:26.553312 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:26.553313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:26.553313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:26.553406 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:26.553599 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:26.553603 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:26.553714 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 15:23:26.553796 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:23:26.553977 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260924 15:23:26.554652 10457 heartbeater.cc:347] Connected to a master server at 127.4.78.126:32969
I20260924 15:23:26.554725 10457 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:26.554911 10457 heartbeater.cc:511] Master 127.4.78.126:32969 requested a full tablet report, sending...
I20260924 15:23:26.555266 9991 ts_manager.cc:194] Registered new tserver with Master: dd72f76b5e4f4768a5058696fdc291fd (127.4.78.67:46381)
I20260924 15:23:26.555611 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:54401
I20260924 15:23:26.563392 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/test-admin.keytab.
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.578938 4409 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 15:23:26.580508 10465 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:26.579774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44114 (local address 127.4.78.65:40793)
0924 15:23:26.579932 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.579937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.579946 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:26.579971 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.579973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.579986 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.580078 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.580428 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44114: BlockingRecv error: recv got EOF from 127.0.0.1:44114 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
W20260924 15:23:26.580595 4409 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5cc' of type FILE
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.581542 4409 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.582410 4409 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:26.590297 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.582891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44116 (local address 127.4.78.65:40793)
0924 15:23:26.582947 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.582949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.582965 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:23:26.583006 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.583007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.583016 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.583074 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.583451 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.584182 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.584881 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.585228 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.585287 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.587331 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:26.587338 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:26.587338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:26.587345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:26.589168 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.589472 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.589474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.589475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.589536 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.589713 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.589714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.589714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.589896 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:26.589999 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:26.590081 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:23:26.590140 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 15:23:26.592033 10465 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:26.591516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44130 (local address 127.4.78.65:40793)
0924 15:23:26.591580 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.591582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.591590 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:26.591610 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.591610 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.591620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.591674 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.591987 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44130: BlockingRecv error: recv got EOF from 127.0.0.1:44130 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 15:23:26.592092 4409 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5cc' of type FILE
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.593003 4409 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.593854 4409 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:26.600440 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.594338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44140 (local address 127.4.78.65:40793)
0924 15:23:26.594400 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.594402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.594453 (+ 51us) server_negotiation.cc:407] Connection header received
0924 15:23:26.594526 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.594527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.594539 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.594610 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.594984 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.595479 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.596181 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.596373 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.596462 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.598435 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:26.598436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:26.598436 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:26.598442 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:26.599561 (+ 1119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.599858 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.599859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.599860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.599910 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.600075 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.600076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.600076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.600165 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:26.600231 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:26.600329 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 15:23:26.600383 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 15:23:26.602116 10465 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:26.601408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44154 (local address 127.4.78.65:40793)
0924 15:23:26.601443 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.601445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.601544 (+ 99us) server_negotiation.cc:407] Connection header received
0924 15:23:26.601624 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.601625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.601634 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.601682 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.602060 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44154: BlockingRecv error: recv got EOF from 127.0.0.1:44154 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 15:23:26.602195 4409 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263382361574-4409-0/krb5kdc/krb5cc' of type FILE
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.603121 4409 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:26.603924 4409 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[9945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:23:26.610586 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:26.604472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44158 (local address 127.4.78.65:40793)
0924 15:23:26.604505 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:26.604507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:26.604535 (+ 28us) server_negotiation.cc:407] Connection header received
0924 15:23:26.604607 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:26.604608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:26.604616 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:26.604679 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:26.605021 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.605466 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.606264 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:26.606463 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:26.606534 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:26.608406 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:26.608407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:26.608407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:26.608414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:26.609487 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.609865 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.609866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.609866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.609912 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:26.610118 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:26.610119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:26.610120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:26.610229 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:26.610325 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:26.610429 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 15:23:26.610493 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:23:26.611375 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 10054
I20260924 15:23:26.616429 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 10190
I20260924 15:23:26.620672 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 10326
I20260924 15:23:26.624998 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 9958
2026-09-24T15:23:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1829 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[10479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[10479](info): set up 2 sockets
Sep 24 15:23:26 dist-test-slave-l766 krb5kdc[10479](info): commencing operation
krb5kdc: starting...
W20260924 15:23:27.681174 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:27 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.126.keytab.
Entry for principal oryx/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.126.keytab.
I20260924 15:23:27.796834 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39321
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.126.keytab
--principal=oryx/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:27.876785 10492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:27.876973 10492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:27.876996 10492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:27.878142 10492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:27.878180 10492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:27.878194 10492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:27.878207 10492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:27.878219 10492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:27.879724 10492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.126.keytab
--principal=oryx/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:27.879962 10492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:27.880192 10492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:27.884975 10498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:27.885025 10497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:27.884984 10500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:27.885165 10492 server_base.cc:1034] running on GCE node
I20260924 15:23:27.885409 10492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:27.885712 10492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:27.886852 10492 hybrid_clock.cc:648] HybridClock initialized: now 1790263407886839 us; error 44 us; skew 500 ppm
Sep 24 15:23:27 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263407, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:27.889240 10492 init.cc:377] Logged in from keytab as oryx/127.4.78.126@KRBTEST.COM (short username oryx)
I20260924 15:23:27.889705 10492 webserver.cc:460] Webserver started at http://127.4.78.126:45055/ using document root <none> and password file <none>
I20260924 15:23:27.889974 10492 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:27.890021 10492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:27.890162 10492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:27.891242 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "4b363fdc18bf4c75b90080f5bb7205db"
format_stamp: "Formatted at 2026-09-24 15:23:27 on dist-test-slave-l766"
server_key: "5b57da99b22c7ea182a762d384a70826"
server_key_iv: "e4f503a6e4cdf893fc793d1862737086"
server_key_version: "encryptionkey@0"
I20260924 15:23:27.891611 10492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "4b363fdc18bf4c75b90080f5bb7205db"
format_stamp: "Formatted at 2026-09-24 15:23:27 on dist-test-slave-l766"
server_key: "5b57da99b22c7ea182a762d384a70826"
server_key_iv: "e4f503a6e4cdf893fc793d1862737086"
server_key_version: "encryptionkey@0"
I20260924 15:23:27.892920 10492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:27.893679 10507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:27.893879 10492 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:27.893967 10492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "4b363fdc18bf4c75b90080f5bb7205db"
format_stamp: "Formatted at 2026-09-24 15:23:27 on dist-test-slave-l766"
server_key: "5b57da99b22c7ea182a762d384a70826"
server_key_iv: "e4f503a6e4cdf893fc793d1862737086"
server_key_version: "encryptionkey@0"
I20260924 15:23:27.894045 10492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:27.906056 10492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:27.907876 10492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:27.908034 10492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:27.912076 10492 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:39321
I20260924 15:23:27.912117 10560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:39321 every 8 connection(s)
I20260924 15:23:27.912465 10492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:27.913456 10561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:27.915694 10561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: Bootstrap starting.
I20260924 15:23:27.916250 10561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:27.916525 10561 log.cc:824] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:27.917205 10561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: No bootstrap required, opened a new log
I20260924 15:23:27.918830 10561 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } }
I20260924 15:23:27.918965 10561 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:27.918987 10561 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b363fdc18bf4c75b90080f5bb7205db, State: Initialized, Role: FOLLOWER
I20260924 15:23:27.919154 10561 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } }
I20260924 15:23:27.919215 10561 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:27.919252 10561 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:27.919299 10561 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:27.920060 10561 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } }
I20260924 15:23:27.920176 10561 leader_election.cc:302] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b363fdc18bf4c75b90080f5bb7205db; no voters:
I20260924 15:23:27.920338 10561 leader_election.cc:288] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:27.920398 10564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:27.920560 10564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 1 LEADER]: Becoming Leader. State: Replica: 4b363fdc18bf4c75b90080f5bb7205db, State: Running, Role: LEADER
I20260924 15:23:27.920590 10561 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:27.920651 10564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } }
I20260924 15:23:27.921072 10566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b363fdc18bf4c75b90080f5bb7205db. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } }
I20260924 15:23:27.921180 10566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:27.921051 10565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } }
I20260924 15:23:27.921258 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 10492
I20260924 15:23:27.921290 10565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:27.921348 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:27.921537 4409 external_mini_cluster.cc:1428] Setting key 717df0b39806548ba88d48f9ae8d220c
I20260924 15:23:27.922313 10572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:27.922827 10572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:27.924561 10572 catalog_manager.cc:1347] Generated new cluster ID: e330cec1d2cb47888aa35da6e67637f9
I20260924 15:23:27.924610 10572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:23:27 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:27.933194 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:27.921972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.4.78.126:39321)
0924 15:23:27.922315 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:27.922325 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:27.922357 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:23:27.922472 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:27.922478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:27.922534 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:27.923231 (+ 697us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:27.923456 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:27.924773 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:27.925796 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:27.926438 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:27.926549 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:27.929294 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:27.929298 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:27.929298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:27.929309 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:27.931608 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:27.931995 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:27.931998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:27.931998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:27.932114 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:27.932322 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:27.932322 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:27.932323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:27.932605 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:27.932754 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:27.932855 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 15:23:27.932979 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":89,"threads_started":1}
I20260924 15:23:27.945549 10572 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:27.946269 10572 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:27.953442 10572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: Generated new TSK 0
I20260924 15:23:27.953601 10572 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab.
Entry for principal oryx/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab.
I20260924 15:23:27.978448 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39321
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab
--principal=oryx/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:28.058759 10588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:28.058986 10588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:28.059022 10588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:28.060137 10588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:28.060201 10588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:28.060242 10588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:28.061708 10588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab
--principal=oryx/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:28.062011 10588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:28.062240 10588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:28.067821 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.067822 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.067881 10593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.068130 10588 server_base.cc:1034] running on GCE node
I20260924 15:23:28.068324 10588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:28.068563 10588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:28.069702 10588 hybrid_clock.cc:648] HybridClock initialized: now 1790263408069688 us; error 28 us; skew 500 ppm
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:28.072044 10588 init.cc:377] Logged in from keytab as oryx/127.4.78.65@KRBTEST.COM (short username oryx)
I20260924 15:23:28.072495 10588 webserver.cc:460] Webserver started at http://127.4.78.65:42723/ using document root <none> and password file <none>
I20260924 15:23:28.072728 10588 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.072798 10588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.072947 10588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:28.074002 10588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "c437b618abc1229fc875e8753f925364"
server_key_iv: "3a4ea3a16c1fb56c9e7e74c1b73d3262"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.074354 10588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "c437b618abc1229fc875e8753f925364"
server_key_iv: "3a4ea3a16c1fb56c9e7e74c1b73d3262"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.075631 10588 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:28.076303 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.076475 10588 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.076577 10588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "c437b618abc1229fc875e8753f925364"
server_key_iv: "3a4ea3a16c1fb56c9e7e74c1b73d3262"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.076648 10588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:28.088373 10588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:28.090175 10588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:28.090338 10588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:28.090554 10588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:28.090850 10588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:28.090904 10588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.090945 10588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:28.090973 10588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.096904 10588 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:37007
I20260924 15:23:28.096951 10718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:37007 every 8 connection(s)
I20260924 15:23:28.097246 10588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:28.103035 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 10588
I20260924 15:23:28.103150 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:28.103328 4409 external_mini_cluster.cc:1428] Setting key ee1d9c3281eb08b5e25fc25f15b8794e
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:28.106841 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.097976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:38317 (local address 127.4.78.126:39321)
0924 15:23:28.098040 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.098042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.098280 (+ 238us) server_negotiation.cc:407] Connection header received
0924 15:23:28.098795 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.098796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.098812 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.098874 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.099805 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.100337 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.101360 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.101589 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.101676 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.103909 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.103910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.103911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.103919 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.105420 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.105859 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.105861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.105861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.105925 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.106181 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.106182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.106183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.106303 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.106380 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.106654 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.106748 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:23:28.107018 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.097981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:38317)
0924 15:23:28.098167 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.098202 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.098692 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.098949 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.098952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.099517 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.099707 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.099712 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.100435 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.100436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.101266 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.101269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.101682 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.101683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.101803 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.102329 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.102337 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.103807 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.105524 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.105531 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.105533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.105755 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.105990 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.105991 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.105991 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.106111 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.106371 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.106378 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.106520 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.106631 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.106785 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":88,"threads_started":1}
I20260924 15:23:28.107512 10719 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39321
I20260924 15:23:28.107623 10719 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:28.107820 10719 heartbeater.cc:511] Master 127.4.78.126:39321 requested a full tablet report, sending...
I20260924 15:23:28.108322 10524 ts_manager.cc:194] Registered new tserver with Master: 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007)
I20260924 15:23:28.108930 10524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:38317
WARNING: no policy specified for oryx/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab.
Entry for principal oryx/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab.
I20260924 15:23:28.127697 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39321
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab
--principal=oryx/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:28.206699 10724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:28.206910 10724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:28.206933 10724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:28.208102 10724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:28.208148 10724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:28.208173 10724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:28.209724 10724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab
--principal=oryx/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:28.209975 10724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:28.210165 10724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:28.214998 10729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.215005 10732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.215054 10724 server_base.cc:1034] running on GCE node
W20260924 15:23:28.215010 10730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.215361 10724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:28.215598 10724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:28.216763 10724 hybrid_clock.cc:648] HybridClock initialized: now 1790263408216750 us; error 30 us; skew 500 ppm
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:28.222793 10724 init.cc:377] Logged in from keytab as oryx/127.4.78.66@KRBTEST.COM (short username oryx)
I20260924 15:23:28.223320 10724 webserver.cc:460] Webserver started at http://127.4.78.66:35133/ using document root <none> and password file <none>
I20260924 15:23:28.223541 10724 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.223584 10724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.223726 10724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:28.224762 10724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "a2d1ee0740124cdbbb87a5f37d1854ea"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "0174fb320cc0c19916a0df7318286724"
server_key_iv: "3522e4486a2b042cd483809e01b4fea4"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.225135 10724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "a2d1ee0740124cdbbb87a5f37d1854ea"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "0174fb320cc0c19916a0df7318286724"
server_key_iv: "3522e4486a2b042cd483809e01b4fea4"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.226508 10724 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:23:28.227206 10739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.227375 10724 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:28.227478 10724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "a2d1ee0740124cdbbb87a5f37d1854ea"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "0174fb320cc0c19916a0df7318286724"
server_key_iv: "3522e4486a2b042cd483809e01b4fea4"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.227550 10724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:28.244565 10724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:28.246905 10724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:28.247085 10724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:28.247335 10724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:28.247627 10724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:28.247677 10724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.247709 10724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:28.247735 10724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.253165 10724 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:46657
I20260924 15:23:28.253235 10854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:46657 every 8 connection(s)
I20260924 15:23:28.253538 10724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:28.261785 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.254270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:33197 (local address 127.4.78.126:39321)
0924 15:23:28.254336 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.254338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.254503 (+ 165us) server_negotiation.cc:407] Connection header received
0924 15:23:28.254942 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.254943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.254952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.255008 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.255849 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.256327 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.257302 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.257479 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.257526 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.259529 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.259531 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.259531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.259537 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.260684 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.261026 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.261027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.261027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.261073 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.261286 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.261286 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.261286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.261378 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.261439 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.261660 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.261720 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:23:28.262059 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 10724
I20260924 15:23:28.262138 10857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.254226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.66:33197)
0924 15:23:28.254372 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.254423 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.254865 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.255087 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.255091 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.255601 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.255770 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.255774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.256399 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.256400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.257232 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.257234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.257546 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.257547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.257636 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.258143 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.258151 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.259441 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.260751 (+ 1310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.260757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.260759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.260967 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.261125 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.261125 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.261125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.261232 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.261439 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.261443 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.261545 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.261638 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.261756 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260924 15:23:28.262179 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:28.262363 4409 external_mini_cluster.cc:1428] Setting key 2b5ed11826eaebb33c8af55932024d0e
I20260924 15:23:28.262768 10855 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39321
I20260924 15:23:28.262877 10855 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:28.263044 10855 heartbeater.cc:511] Master 127.4.78.126:39321 requested a full tablet report, sending...
I20260924 15:23:28.263447 10524 ts_manager.cc:194] Registered new tserver with Master: a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657)
I20260924 15:23:28.263860 10524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:33197
WARNING: no policy specified for oryx/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab.
Entry for principal oryx/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab.
I20260924 15:23:28.287009 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39321
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab
--principal=oryx/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:28.380687 10860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:28.380929 10860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:28.380955 10860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:28.382205 10860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:28.382246 10860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:28.382272 10860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:28.383698 10860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab
--principal=oryx/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:28.383934 10860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:28.384161 10860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:28.388898 10865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.388936 10866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.388968 10860 server_base.cc:1034] running on GCE node
W20260924 15:23:28.388909 10868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.389267 10860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:28.389499 10860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:28.390630 10860 hybrid_clock.cc:648] HybridClock initialized: now 1790263408390620 us; error 26 us; skew 500 ppm
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:28.392922 10860 init.cc:377] Logged in from keytab as oryx/127.4.78.67@KRBTEST.COM (short username oryx)
I20260924 15:23:28.393333 10860 webserver.cc:460] Webserver started at http://127.4.78.67:43095/ using document root <none> and password file <none>
I20260924 15:23:28.393566 10860 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.393633 10860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.393787 10860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:28.394815 10860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "fee2270f7c7647f89b868163f951feb2"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "37573384f06d57b440a071ac6299b046"
server_key_iv: "b20c7b9927fe2c7a3f1d62cd82fe62f6"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.395171 10860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "fee2270f7c7647f89b868163f951feb2"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "37573384f06d57b440a071ac6299b046"
server_key_iv: "b20c7b9927fe2c7a3f1d62cd82fe62f6"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.396536 10860 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:23:28.397289 10875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.397493 10860 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:28.397610 10860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "fee2270f7c7647f89b868163f951feb2"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "37573384f06d57b440a071ac6299b046"
server_key_iv: "b20c7b9927fe2c7a3f1d62cd82fe62f6"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.397686 10860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:28.442294 10860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:28.444207 10860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:28.444401 10860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:28.444651 10860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:28.444977 10860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:28.445031 10860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.445081 10860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:28.445108 10860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.450999 10860 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:39383
I20260924 15:23:28.451054 10988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:39383 every 8 connection(s)
I20260924 15:23:28.451411 10860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:28.451754 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 10860
I20260924 15:23:28.451869 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:28.452096 4409 external_mini_cluster.cc:1428] Setting key 1d7d19aeda477d9e6a8a5b8648b39a6c
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:28.460361 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.452304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:54963 (local address 127.4.78.126:39321)
0924 15:23:28.452355 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.452357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.452500 (+ 143us) server_negotiation.cc:407] Connection header received
0924 15:23:28.453122 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.453123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.453133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.453179 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.454267 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.454766 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.455770 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.455946 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.455995 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.458064 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.458065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.458066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.458072 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.459237 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.459610 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.459611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.459611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.459658 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.459861 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.459862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.459862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.459955 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.460003 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.460230 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.460299 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:23:28.460561 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.452119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:54963)
0924 15:23:28.452367 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.452407 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.453032 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.453232 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.453235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.453975 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.454185 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.454190 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.454868 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.454869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.455691 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.455693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.456017 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.456018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.456158 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.456562 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.456570 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.457984 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.459322 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.459328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.459330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.459546 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.459715 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.459716 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.459716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.459809 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.460009 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.460013 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.460122 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.460201 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.460311 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":81,"threads_started":1}
I20260924 15:23:28.461040 10989 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39321
I20260924 15:23:28.461143 10989 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:28.461313 10989 heartbeater.cc:511] Master 127.4.78.126:39321 requested a full tablet report, sending...
I20260924 15:23:28.461719 10524 ts_manager.cc:194] Registered new tserver with Master: fee2270f7c7647f89b868163f951feb2 (127.4.78.67:39383)
I20260924 15:23:28.462105 10524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:54963
I20260924 15:23:28.463699 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.78.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.125.keytab.
Entry for principal oryx/127.4.78.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.125.keytab.
I20260924 15:23:28.487936 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.125:37925
--webserver_interface=127.4.78.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.125.keytab
--principal=oryx/127.4.78.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39321,127.4.78.125:37925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:28.568369 10994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:28.568550 10994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:28.568590 10994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:28.569718 10994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:28.569759 10994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:28.569773 10994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:28.569785 10994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:28.569855 10994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:28.571333 10994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39321,127.4.78.125:37925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.125.keytab
--principal=oryx/127.4.78.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.125:37925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.78.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
W20260924 15:23:28.571455 10994 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.78.126:39321,127.4.78.125:37925'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 15:23:28.571595 10994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:28.571811 10994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:28.576442 11002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.576481 11000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.576453 10999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.576752 10994 server_base.cc:1034] running on GCE node
I20260924 15:23:28.576922 10994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:28.577144 10994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:28.578303 10994 hybrid_clock.cc:648] HybridClock initialized: now 1790263408578288 us; error 23 us; skew 500 ppm
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:28.580768 10994 init.cc:377] Logged in from keytab as oryx/127.4.78.125@KRBTEST.COM (short username oryx)
I20260924 15:23:28.581250 10994 webserver.cc:460] Webserver started at http://127.4.78.125:38639/ using document root <none> and password file <none>
I20260924 15:23:28.581508 10994 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.581575 10994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.581725 10994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:28.582726 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data/instance:
uuid: "41f23ec6a5ce40208366634772d324b1"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "aa146c5ae66de9dd2beeceebdf2224b8"
server_key_iv: "bda86f4d80835371dcff246cbf0d352f"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.583060 10994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal/instance:
uuid: "41f23ec6a5ce40208366634772d324b1"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "aa146c5ae66de9dd2beeceebdf2224b8"
server_key_iv: "bda86f4d80835371dcff246cbf0d352f"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.584296 10994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:23:28.584997 11009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.585153 10994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:28.585239 10994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
uuid: "41f23ec6a5ce40208366634772d324b1"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "aa146c5ae66de9dd2beeceebdf2224b8"
server_key_iv: "bda86f4d80835371dcff246cbf0d352f"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.585315 10994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:28.597945 10994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:28.599659 10994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:28.599810 10994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:28.603737 10994 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.125:37925
I20260924 15:23:28.603771 11061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.125:37925 every 8 connection(s)
I20260924 15:23:28.604139 10994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
I20260924 15:23:28.605042 11062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:28.606539 11062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.125@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:28.612052 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 10994
I20260924 15:23:28.615623 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.607146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.4.78.126:39321)
0924 15:23:28.607191 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.607193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.607392 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:23:28.607888 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.607889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.607902 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.607970 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.608543 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.609070 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.610109 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.610323 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.610400 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.612386 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.612387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.612388 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.612395 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.614275 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.614640 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.614641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.614641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.614704 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.614997 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.614998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.614998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.615139 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.615192 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.615490 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.615550 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:23:28.615846 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.607131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:40858)
0924 15:23:28.607278 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.607308 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.607775 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.608038 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.608042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.608267 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.608444 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.608448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.609164 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.609165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.610023 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.610025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.610406 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.610407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.610522 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.610924 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.610932 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.612292 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.614361 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.614366 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.614368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.614571 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.614776 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.614777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.614777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.614910 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.615208 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.615214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.615361 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.615469 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.615616 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260924 15:23:28.616499 11062 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.78.125@KRBTEST.COM
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.125@KRBTEST.COM for oryx/127.4.78.125@KRBTEST.COM
I20260924 15:23:28.622555 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.612534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.4.78.125:37925)
0924 15:23:28.612702 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.612706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.612851 (+ 145us) server_negotiation.cc:407] Connection header received
0924 15:23:28.612874 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.612876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.612886 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.613064 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.613646 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.614646 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.615681 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.616179 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.616241 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.619045 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.619051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.619052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.619057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.620748 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.621156 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.621159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.621159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.621489 (+ 330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.621713 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.621714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.621715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.621910 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.622087 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.622167 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.622245 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":76,"threads_started":1}
I20260924 15:23:28.624013 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.616733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.0.0.1:45406)
0924 15:23:28.616779 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.616794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.616846 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.617069 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.617070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.617174 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.617222 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.617223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.618140 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.618141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.618846 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.618849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.619261 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.619262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.619339 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.619594 (+ 255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.619599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.620954 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.623182 (+ 2228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.623183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.623184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.623418 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.623582 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.623582 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.623583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.623631 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.623820 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.623820 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.623849 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.623894 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.623960 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 15:23:28.624122 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.616779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45406 (local address 127.4.78.125:37925)
0924 15:23:28.616950 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.616956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.616964 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:28.616984 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.616985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.616996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.617065 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.617292 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.618070 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.618923 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.619191 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.619238 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.621031 (+ 1793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.621032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.621032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.621037 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.623085 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.623471 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.623472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.623472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.623523 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.623667 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.623668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.623668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.623748 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.623831 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.623912 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.624004 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260924 15:23:28.625280 11062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Bootstrap starting.
I20260924 15:23:28.625926 11062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:28.626194 11062 log.cc:824] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:28.626857 11062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: No bootstrap required, opened a new log
I20260924 15:23:28.628166 11062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } }
I20260924 15:23:28.628306 11062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:28.628350 11062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41f23ec6a5ce40208366634772d324b1, State: Initialized, Role: FOLLOWER
I20260924 15:23:28.628506 11062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } }
I20260924 15:23:28.628786 11062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:28.628757 11069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } } }
I20260924 15:23:28.628876 11069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:23:28.630422 11081 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:23:28.630486 11081 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:23:28.632037 10994 master_runner.cc:428] Detected that this master 41f23ec6a5ce40208366634772d324b1 is joining an existing cluster
I20260924 15:23:28.632090 10994 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.78.125:37925
I20260924 15:23:28.639760 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.632941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.0.0.1:45418)
0924 15:23:28.633114 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.633124 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.633396 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.633540 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.633541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.633732 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.633788 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.633789 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.634675 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.634676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.635248 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.635249 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.635613 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.635614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.635671 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.636087 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.636092 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.636456 (+ 364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.638616 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.638617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.638617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.638832 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.639077 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.639078 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.639078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.639143 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.639417 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.639418 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.639460 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.639542 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.639641 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":2,"spinlock_wait_cycles":7936,"thread_start_us":136,"threads_started":1}
I20260924 15:23:28.639873 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.633181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45418 (local address 127.4.78.125:37925)
0924 15:23:28.633217 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.633218 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.633225 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:28.633469 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.633470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.633478 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.633536 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.633873 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.634591 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.635345 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.635544 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.635615 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.636554 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.636555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.636556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.636562 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.638521 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.638919 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.638920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.638920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.638988 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.639213 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.639214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.639214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.639333 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.639396 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.639580 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.639701 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":23}
I20260924 15:23:28.640650 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.632927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:40866)
0924 15:23:28.633096 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.633108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.633160 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.633362 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.633363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.633570 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.633628 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.633630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.634590 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.634591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.635554 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.635557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.636079 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.636079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.636142 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.636518 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.636523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.636908 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.639378 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.639378 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.639379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.639576 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.639793 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.639794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.639794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.640050 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.640314 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.640315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.640353 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.640446 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.640545 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":37,"thread_start_us":102,"threads_started":1}
I20260924 15:23:28.640691 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.633182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.4.78.126:39321)
0924 15:23:28.633256 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.633258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.633265 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:28.633286 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.633287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.633296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.633359 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.633699 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.634504 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.635657 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.635964 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.636005 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.636981 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.636983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.636983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.636990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.639296 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.639661 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.639663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.639663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.639718 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.640121 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.640122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.640123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.640235 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.640309 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.640485 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.640557 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263407, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.126@KRBTEST.COM for oryx/127.4.78.125@KRBTEST.COM
I20260924 15:23:28.649472 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.642799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.4.78.125:37925)
0924 15:23:28.642833 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.642835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.642944 (+ 109us) server_negotiation.cc:407] Connection header received
0924 15:23:28.643027 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.643028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.643034 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.643076 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.643515 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.643919 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.644951 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.645110 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.645153 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.647070 (+ 1917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.647072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.647072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.647078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.648328 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.648693 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.648694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.648694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.648749 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.648946 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.648947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.648947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.649055 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.649126 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.649319 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.649386 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:28.649523 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.642704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.0.0.1:45432)
0924 15:23:28.642847 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.642893 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.642961 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.643082 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.643084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.643392 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.643445 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.643447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.643989 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.643990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.644872 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.644874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.645173 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.645174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.645255 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.645711 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.645716 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.646984 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.648413 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.648420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.648423 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.648618 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.648818 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.648819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.648819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.648875 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.649133 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.649140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.649188 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.649273 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.649390 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260924 15:23:28.650018 10524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.78.125:37925: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "41f23ec6a5ce40208366634772d324b1"
member_type: NON_VOTER
last_known_addr {
host: "127.4.78.125"
port: 37925
}
attrs {
promote: true
}
}
dest_uuid: "4b363fdc18bf4c75b90080f5bb7205db"
cas_config_opid_index: -1
I20260924 15:23:28.650327 10524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } }
I20260924 15:23:28.650753 11091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 41f23ec6a5ce40208366634772d324b1 (127.4.78.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } }
I20260924 15:23:28.651114 11036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:28.651711 11091 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.78.125:37925
I20260924 15:23:28.651926 10994 master.cc:537] Master@127.4.78.125:37925 shutting down...
I20260924 15:23:28.652045 11036 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Refusing update from remote peer 4b363fdc18bf4c75b90080f5bb7205db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 15:23:28.652356 11090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:23:28.653055 10510 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260924 15:23:28.653475 10510 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db -> Peer 41f23ec6a5ce40208366634772d324b1 (127.4.78.125:37925): Couldn't send request to peer 41f23ec6a5ce40208366634772d324b1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 15:23:28.651883 11091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } }
I20260924 15:23:28.653910 11091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:28.654874 10994 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:28.654958 10994 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:28.654996 10994 tablet_replica.cc:330] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: stopping tablet replica
I20260924 15:23:28.667944 10994 master.cc:552] Master@127.4.78.125:37925 shutdown complete.
I20260924 15:23:28.667994 10994 master_runner.cc:305] Clearing existing system tablet
I20260924 15:23:28.668121 10994 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.668166 10994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.668852 10994 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260924 15:23:28.669363 11095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.669510 10994 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:28.669569 10994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.669595 10994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
uuid: "41f23ec6a5ce40208366634772d324b1"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "aa146c5ae66de9dd2beeceebdf2224b8"
server_key_iv: "bda86f4d80835371dcff246cbf0d352f"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.670380 10994 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:28.671775 10994 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 15:23:28.671842 10994 log.cc:1196] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 15:23:28.672039 10994 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Deleting consensus metadata
I20260924 15:23:28.672155 10994 master_runner.cc:315] Copying system tablet from 127.4.78.126:39321
I20260924 15:23:28.672739 10994 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: tablet copy: Beginning tablet copy session from remote peer at address 127.4.78.126:39321
I20260924 15:23:28.679006 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.673048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.4.78.126:39321)
0924 15:23:28.673110 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.673112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.673212 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:23:28.673293 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.673294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.673304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.673367 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.673746 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.674328 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.675360 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.675533 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.675581 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.676534 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.676535 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.676535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.676541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.677988 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.678356 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.678357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.678357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.678406 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.678594 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.678595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.678595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.678695 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.678747 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.678894 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.678949 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:23:28.679131 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.673005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:40880)
0924 15:23:28.673134 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.673146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.673211 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.673375 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.673376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.673566 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.673631 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.673633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.674406 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.674407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.675280 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.675283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.675602 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.675603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.675671 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.676083 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.676089 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.676449 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.678059 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.678060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.678061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.678277 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.678465 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.678466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.678466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.678532 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.678759 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.678760 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.678803 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.678879 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.678973 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":88,"threads_started":1}
I20260924 15:23:28.679459 10544 tablet_copy_service.cc:140] P 4b363fdc18bf4c75b90080f5bb7205db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 41f23ec6a5ce40208366634772d324b1 ({username='oryx', principal='oryx/127.4.78.125@KRBTEST.COM'} at 127.0.0.1:40880)
I20260924 15:23:28.679550 10544 tablet_copy_service.cc:161] P 4b363fdc18bf4c75b90080f5bb7205db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 41f23ec6a5ce40208366634772d324b1 at {username='oryx', principal='oryx/127.4.78.125@KRBTEST.COM'} at 127.0.0.1:40880: session id = 41f23ec6a5ce40208366634772d324b1-00000000000000000000000000000000
I20260924 15:23:28.680212 10544 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 15:23:28.680871 10994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 15:23:28.682030 10994 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: tablet copy: Starting download of 0 data blocks...
I20260924 15:23:28.682168 10994 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: tablet copy: Starting download of 1 WAL segments...
I20260924 15:23:28.682909 10994 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 15:23:28.683693 10544 tablet_copy_service.cc:342] P 4b363fdc18bf4c75b90080f5bb7205db: Request end of tablet copy session 41f23ec6a5ce40208366634772d324b1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.78.125@KRBTEST.COM'} at 127.0.0.1:40880
I20260924 15:23:28.683758 10544 tablet_copy_service.cc:433] P 4b363fdc18bf4c75b90080f5bb7205db: ending tablet copy session 41f23ec6a5ce40208366634772d324b1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 41f23ec6a5ce40208366634772d324b1
I20260924 15:23:28.686029 10994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:28.687100 11102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.687120 11103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.687171 11105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.687381 10994 server_base.cc:1034] running on GCE node
I20260924 15:23:28.687487 10994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:28.687616 10994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:28.688741 10994 hybrid_clock.cc:648] HybridClock initialized: now 1790263408688730 us; error 23 us; skew 500 ppm
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:28.690143 10994 init.cc:377] Logged in from keytab as oryx/127.4.78.125@KRBTEST.COM (short username oryx)
I20260924 15:23:28.690379 10994 webserver.cc:460] Webserver started at http://127.4.78.125:42931/ using document root <none> and password file <none>
I20260924 15:23:28.690491 10994 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.690527 10994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.691179 10994 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:28.691704 11112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.691890 10994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:28.691943 10994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
uuid: "41f23ec6a5ce40208366634772d324b1"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "aa146c5ae66de9dd2beeceebdf2224b8"
server_key_iv: "bda86f4d80835371dcff246cbf0d352f"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.691988 10994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:28.702105 10994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:28.703627 10994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:28.706534 10994 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.125:37925
I20260924 15:23:28.706737 11165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.125:37925 every 8 connection(s)
I20260924 15:23:28.706879 10994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
I20260924 15:23:28.707520 11166 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:23:28.707918 11166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Bootstrap starting.
I20260924 15:23:28.710280 11166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:23:28.710489 11166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: Bootstrap complete.
I20260924 15:23:28.710670 11166 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } }
I20260924 15:23:28.710767 11166 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 41f23ec6a5ce40208366634772d324b1, State: Initialized, Role: LEARNER
I20260924 15:23:28.710806 11166 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } }
I20260924 15:23:28.710994 11166 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:28.710986 11169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } }
I20260924 15:23:28.711064 11169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:23:28.712360 11180 catalog_manager.cc:1259] Loaded cluster ID: e330cec1d2cb47888aa35da6e67637f9
I20260924 15:23:28.712417 11180 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: loading cluster ID for follower catalog manager: success
I20260924 15:23:28.713359 11180 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: acquiring CA information for follower catalog manager: success
I20260924 15:23:28.713641 11180 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:23:28.731211 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.724177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.4.78.125:37925)
0924 15:23:28.724302 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.724304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.724312 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:28.724333 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.724334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.724343 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.724413 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.725121 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.725770 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.726803 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.726990 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.727036 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.728408 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.728409 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.728409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.728415 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.729967 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.730342 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.730342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.730343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.730390 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.730596 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.730597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.730597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.730693 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.730780 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.730966 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.731045 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for oryx/127.4.78.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.124.keytab.
Entry for principal oryx/127.4.78.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.124.keytab.
I20260924 15:23:28.754590 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.124:46113
--webserver_interface=127.4.78.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.124.keytab
--principal=oryx/127.4.78.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39321,127.4.78.125:37925,127.4.78.124:46113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:28.837178 11184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:28.837376 11184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:28.837399 11184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:28.838615 11184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:28.838682 11184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:28.838717 11184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:28.838740 11184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:28.838769 11184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:28.840231 11184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39321,127.4.78.125:37925,127.4.78.124:46113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.124.keytab
--principal=oryx/127.4.78.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.124:46113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.78.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:28.840415 11184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:28.840574 11184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:28.845248 11190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.845243 11189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.845240 11192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.845415 11184 server_base.cc:1034] running on GCE node
I20260924 15:23:28.845652 11184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:28.845938 11184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:28.847076 11184 hybrid_clock.cc:648] HybridClock initialized: now 1790263408847064 us; error 26 us; skew 500 ppm
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:28.849587 11184 init.cc:377] Logged in from keytab as oryx/127.4.78.124@KRBTEST.COM (short username oryx)
I20260924 15:23:28.849997 11184 webserver.cc:460] Webserver started at http://127.4.78.124:45343/ using document root <none> and password file <none>
I20260924 15:23:28.850194 11184 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.850245 11184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.850365 11184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:28.851380 11184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data/instance:
uuid: "ce0df8694f9a404ea9292cec4d688550"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "b3cb58b9100e16f4a674ae052834a74c"
server_key_iv: "2911d07db111ef79462ccf635fa3df9a"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.851666 11184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal/instance:
uuid: "ce0df8694f9a404ea9292cec4d688550"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "b3cb58b9100e16f4a674ae052834a74c"
server_key_iv: "2911d07db111ef79462ccf635fa3df9a"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.852880 11184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:28.853520 11199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.853686 11184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:28.853772 11184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
uuid: "ce0df8694f9a404ea9292cec4d688550"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "b3cb58b9100e16f4a674ae052834a74c"
server_key_iv: "2911d07db111ef79462ccf635fa3df9a"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.853885 11184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:28.863960 11184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:28.866209 11184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:28.866351 11184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:28.870056 11184 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.124:46113
I20260924 15:23:28.870086 11251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.124:46113 every 8 connection(s)
I20260924 15:23:28.870393 11184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
I20260924 15:23:28.871279 11252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:28.872666 11252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.124@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:28.878784 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 11184
I20260924 15:23:28.880913 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.873419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40882 (local address 127.4.78.126:39321)
0924 15:23:28.873509 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.873512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.873525 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:23:28.874040 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.874041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.874050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.874108 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.874639 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.875116 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.876021 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.876193 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.876239 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.878089 (+ 1850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.878090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.878091 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.878098 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.879550 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.879999 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.880001 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.880001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.880056 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.880288 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.880289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.880290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.880420 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.880487 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.880749 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.880823 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:23:28.880995 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.873259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:40882)
0924 15:23:28.873412 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.873450 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.873956 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.874173 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.874177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.874399 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.874570 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.874574 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.875186 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.875187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.875953 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.875955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.876257 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.876257 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.876344 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.876727 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.876735 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.878006 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.879625 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.879628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.879630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.879927 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.880122 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.880123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.880123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.880226 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.880511 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.880516 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.880624 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.880706 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.880828 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260924 15:23:28.881515 11252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.78.124@KRBTEST.COM
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.124@KRBTEST.COM for oryx/127.4.78.125@KRBTEST.COM
I20260924 15:23:28.888000 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.879422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53794 (local address 127.4.78.124:46113)
0924 15:23:28.879559 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.879563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.879586 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:23:28.879624 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.879626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.879640 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.879734 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.880355 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.881286 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.882350 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.882772 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.882840 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.885054 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.885059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.885060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.885065 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.886834 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.887154 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.887157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.887157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.887221 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.887412 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.887412 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.887412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.887575 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.887698 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.887794 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.887865 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":85,"threads_started":1}
I20260924 15:23:28.888347 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.881730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.0.0.1:45456)
0924 15:23:28.881788 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.881797 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.881848 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.882027 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.882028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.882165 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.882208 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.882209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.883099 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.883100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.883576 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.883578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.883939 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.883940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.884002 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.884415 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.884419 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.885628 (+ 1209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.887194 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.887195 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.887195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.887416 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.887594 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.887594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.887594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.887671 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.888034 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.888035 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.888098 (+ 63us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.888178 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.888259 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:23:28.888391 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.881845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.4.78.125:37925)
0924 15:23:28.881892 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.881894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.881901 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:28.881928 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.881928 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.881939 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.882024 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.882280 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.882997 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.883654 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.883870 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.883942 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.885713 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.885714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.885714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.885721 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.887103 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.887481 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.887482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.887482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.887532 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.887737 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.887738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.887738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.887865 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.887949 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.888229 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.888309 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:23:28.888799 11252 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.124@KRBTEST.COM for oryx/127.4.78.124@KRBTEST.COM
I20260924 15:23:28.894522 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.889041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.0.0.1:53796)
0924 15:23:28.889088 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.889098 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.889135 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.889255 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.889256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.889365 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.889391 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.889392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.889995 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.889996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.890435 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.890437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.890783 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.890784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.890838 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.891124 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.891127 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.892200 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.893730 (+ 1530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.893731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.893731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.893936 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.894113 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.894114 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.894114 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.894163 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.894327 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.894327 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.894358 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.894404 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.894470 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:23:28.894582 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.889069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53796 (local address 127.4.78.124:46113)
0924 15:23:28.889096 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.889097 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.889126 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:23:28.889196 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.889197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.889204 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.889241 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.889449 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.889935 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.890498 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.890738 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.890783 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.892273 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.892274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.892274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.892280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.893666 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.893988 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.893989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.893989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.894050 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.894204 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.894204 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.894204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.894282 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.894325 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.894417 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.894512 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":11}
I20260924 15:23:28.895691 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Bootstrap starting.
I20260924 15:23:28.896497 11252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:28.896757 11252 log.cc:824] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:28.897419 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: No bootstrap required, opened a new log
I20260924 15:23:28.898916 11252 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } }
I20260924 15:23:28.899029 11252 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:28.899050 11252 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce0df8694f9a404ea9292cec4d688550, State: Initialized, Role: FOLLOWER
I20260924 15:23:28.899130 11252 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } }
I20260924 15:23:28.899364 11252 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:28.899359 11257 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } } }
I20260924 15:23:28.899487 11257 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:23:28.901109 11269 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:23:28.901180 11269 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:23:28.903264 11184 master_runner.cc:428] Detected that this master ce0df8694f9a404ea9292cec4d688550 is joining an existing cluster
I20260924 15:23:28.903313 11184 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.78.124:46113
I20260924 15:23:28.911221 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.904662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53812 (local address 127.4.78.124:46113)
0924 15:23:28.904698 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.904700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.904865 (+ 165us) server_negotiation.cc:407] Connection header received
0924 15:23:28.904943 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.904944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.904951 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.904978 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.905401 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.906080 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907189 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.907483 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907522 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.908567 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.908569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.908569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.908576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.910120 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.910450 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.910451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.910451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.910512 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.910697 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.910698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.910698 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.910842 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.910911 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.911028 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.911138 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":20}
I20260924 15:23:28.911413 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.904539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.0.0.1:53812)
0924 15:23:28.904818 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.904827 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.904883 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.905043 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.905044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.905252 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.905323 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.905324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.906153 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.906153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.907094 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.907096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.907604 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.907663 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.908140 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.908144 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.908476 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.910209 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.910210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.910210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.910380 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.910578 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.910578 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.910579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.910630 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.910899 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.910900 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.910945 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.910999 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.911076 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":94,"thread_start_us":52,"threads_started":1}
I20260924 15:23:28.926057 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.904438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.4.78.125:37925)
0924 15:23:28.904523 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.904525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.904535 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:28.904624 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.904624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.904636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.904707 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.905145 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.905967 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907192 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.907485 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907523 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.923110 (+ 15587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.923111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.923112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.923120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.924839 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.925236 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.925237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.925237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.925296 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.925466 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.925468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.925468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.925611 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.925683 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.925880 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.925966 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 15:23:28.926064 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.904311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.0.0.1:45468)
0924 15:23:28.904487 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.904498 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.904544 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.904712 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.904713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.904977 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.905054 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.905056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.906053 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.906053 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.907094 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.907096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.907605 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.907663 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.922539 (+ 14876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.922544 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.923008 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.924922 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.924923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.924924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.925162 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.925352 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.925352 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.925353 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.925397 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.925691 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.925692 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.925741 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.925836 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.925950 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":35,"thread_start_us":94,"threads_started":1}
I20260924 15:23:28.926100 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.904390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.4.78.126:39321)
0924 15:23:28.904486 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.904488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.904687 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:23:28.904792 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.904792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.904806 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.904867 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.905328 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.906130 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907205 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.907496 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907537 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.922994 (+ 15457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.922996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.922996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.923004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.924693 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.925097 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.925099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.925099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.925155 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.925522 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.925523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.925523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.925643 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.925702 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.925934 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.926004 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 15:23:28.926491 11276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.904319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:40886)
0924 15:23:28.904632 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.904641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.904709 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.904914 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.904914 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.905168 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.905234 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.905236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.906227 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.906228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.907104 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.907107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.907612 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.907613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.907696 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.922362 (+ 14666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.922368 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.922856 (+ 488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.924785 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.924787 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.924787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.925016 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.925377 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.925378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.925378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.925435 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.925730 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.925731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.925780 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.926213 (+ 433us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.926344 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":95,"threads_started":1}
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263407, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.126@KRBTEST.COM for oryx/127.4.78.124@KRBTEST.COM
I20260924 15:23:28.934827 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.928329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.0.0.1:53820)
0924 15:23:28.928373 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.928382 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.928415 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.928563 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.928563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.928724 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.928770 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.928772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.929365 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.929366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.930243 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.930245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.930644 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.930645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.930722 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.931040 (+ 318us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.931045 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.932298 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.933830 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.933831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.933832 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.933998 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.934197 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.934198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.934198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.934253 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.934518 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.934519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.934558 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.934649 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.934739 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 15:23:28.934937 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.928403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.4.78.124:46113)
0924 15:23:28.928438 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.928440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.928449 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:28.928481 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.928481 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.928496 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.928541 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.928836 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.929290 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.930316 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.930570 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.930637 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.932387 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.932393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.932393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.932399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.933691 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.934079 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.934080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.934081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.934133 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.934322 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.934323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.934323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.934442 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.934513 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.934672 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.934739 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:23:28.936372 10524 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.78.124:46113: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ce0df8694f9a404ea9292cec4d688550"
member_type: NON_VOTER
last_known_addr {
host: "127.4.78.124"
port: 46113
}
attrs {
promote: true
}
}
dest_uuid: "4b363fdc18bf4c75b90080f5bb7205db"
cas_config_opid_index: 5
I20260924 15:23:28.936606 10524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } }
I20260924 15:23:28.936944 11091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ce0df8694f9a404ea9292cec4d688550 (127.4.78.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.937332 11226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:28.937948 11091 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.78.124:46113
I20260924 15:23:28.938030 11090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.938122 11090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:28.938261 11184 master.cc:537] Master@127.4.78.124:46113 shutting down...
I20260924 15:23:28.938344 11226 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Refusing update from remote peer 4b363fdc18bf4c75b90080f5bb7205db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 15:23:28.938707 11090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:23:28.939445 10510 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db -> Peer ce0df8694f9a404ea9292cec4d688550 (127.4.78.124:46113): Couldn't send request to peer ce0df8694f9a404ea9292cec4d688550. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 15:23:28.942480 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.937035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.0.0.1:45474)
0924 15:23:28.937158 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.937168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.937206 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.938447 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.938448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.938449 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:28.938484 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.938486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.940147 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.940148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.941370 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.941372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.942230 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.942231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.942291 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.942312 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.942360 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.942400 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 15:23:28.942704 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.938094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45474 (local address 127.4.78.125:37925)
0924 15:23:28.938269 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.938271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.938279 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:28.938301 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.938302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.938325 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.938390 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:28.938585 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.940057 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.941462 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.942064 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.942125 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.942198 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.942604 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.942636 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260924 15:23:28.942878 11139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 LEARNER]: Refusing update from remote peer 4b363fdc18bf4c75b90080f5bb7205db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 15:23:28.943041 11090 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 15:23:28.943564 11139 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ce0df8694f9a404ea9292cec4d688550 (127.4.78.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.944399 11184 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:28.944492 11184 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:28.944547 11184 tablet_replica.cc:330] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: stopping tablet replica
I20260924 15:23:28.944840 11169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b363fdc18bf4c75b90080f5bb7205db. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.944923 11169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:23:28.944957 11169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: NON_VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: true } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.945003 11169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:23:28.945469 11090 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: attempting to promote NON_VOTER 41f23ec6a5ce40208366634772d324b1 to VOTER
I20260924 15:23:28.945638 11090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } }
I20260924 15:23:28.946231 11139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 LEARNER]: Refusing update from remote peer 4b363fdc18bf4c75b90080f5bb7205db: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 15:23:28.946529 11091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 15:23:28.946758 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.946621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.0.0.1:53832)
0924 15:23:28.946697 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.946735 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:46113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 15:23:28.946832 10510 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db -> Peer ce0df8694f9a404ea9292cec4d688550 (127.4.78.124:46113): Couldn't send request to peer ce0df8694f9a404ea9292cec4d688550. Status: Network error: Client connection negotiation failed: client connection to 127.4.78.124:46113: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 15:23:28.947077 11091 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 41f23ec6a5ce40208366634772d324b1 (127.4.78.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.947839 11091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.947911 11091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:28.947877 11280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.947966 11280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:28.948140 11139 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 41f23ec6a5ce40208366634772d324b1 (127.4.78.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.948828 11279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:28.948957 11279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:23:28.957482 11184 master.cc:552] Master@127.4.78.124:46113 shutdown complete.
I20260924 15:23:28.957531 11184 master_runner.cc:305] Clearing existing system tablet
I20260924 15:23:28.957659 11184 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.957713 11184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.958436 11184 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.958962 11282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.959111 11184 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:28.959172 11184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:28.959197 11184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
uuid: "ce0df8694f9a404ea9292cec4d688550"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "b3cb58b9100e16f4a674ae052834a74c"
server_key_iv: "2911d07db111ef79462ccf635fa3df9a"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.959916 11184 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:28.961402 11184 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 15:23:28.961470 11184 log.cc:1196] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 15:23:28.961655 11184 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Deleting consensus metadata
I20260924 15:23:28.961920 11184 master_runner.cc:315] Copying system tablet from 127.4.78.126:39321
I20260924 15:23:28.962563 11184 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: tablet copy: Beginning tablet copy session from remote peer at address 127.4.78.126:39321
I20260924 15:23:28.968571 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.962969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.4.78.126:39321)
0924 15:23:28.963014 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:28.963016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:28.963029 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:23:28.963094 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:28.963095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:28.963102 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:28.963144 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:28.963482 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.963944 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.964979 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.965151 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.965199 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.966212 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:28.966213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:28.966214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:28.966220 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:28.967518 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.967923 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.967924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.967924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.967973 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:28.968157 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:28.968158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:28.968158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:28.968247 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:28.968304 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:28.968455 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 15:23:28.968511 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:28.968753 11288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:28.962846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:40888)
0924 15:23:28.962973 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:28.962984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:28.963040 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:28.963159 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:28.963159 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:28.963348 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:28.963416 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.963417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.964019 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.964020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.964899 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:28.964902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:28.965218 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:28.965219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:28.965286 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:28.965704 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:28.965709 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:28.966122 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:28.967598 (+ 1476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.967599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.967600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.967833 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.968036 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:28.968037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:28.968037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:28.968099 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:28.968314 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:28.968315 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:28.968360 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 15:23:28.968435 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:23:28.968533 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":82,"threads_started":1}
I20260924 15:23:28.968981 10544 tablet_copy_service.cc:140] P 4b363fdc18bf4c75b90080f5bb7205db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ce0df8694f9a404ea9292cec4d688550 ({username='oryx', principal='oryx/127.4.78.124@KRBTEST.COM'} at 127.0.0.1:40888)
I20260924 15:23:28.969053 10544 tablet_copy_service.cc:161] P 4b363fdc18bf4c75b90080f5bb7205db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ce0df8694f9a404ea9292cec4d688550 at {username='oryx', principal='oryx/127.4.78.124@KRBTEST.COM'} at 127.0.0.1:40888: session id = ce0df8694f9a404ea9292cec4d688550-00000000000000000000000000000000
I20260924 15:23:28.969477 10544 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 15:23:28.970019 11184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 15:23:28.970861 11184 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: tablet copy: Starting download of 0 data blocks...
I20260924 15:23:28.971020 11184 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: tablet copy: Starting download of 1 WAL segments...
I20260924 15:23:28.971733 11184 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 15:23:28.972481 10544 tablet_copy_service.cc:342] P 4b363fdc18bf4c75b90080f5bb7205db: Request end of tablet copy session ce0df8694f9a404ea9292cec4d688550-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.78.124@KRBTEST.COM'} at 127.0.0.1:40888
I20260924 15:23:28.972539 10544 tablet_copy_service.cc:433] P 4b363fdc18bf4c75b90080f5bb7205db: ending tablet copy session ce0df8694f9a404ea9292cec4d688550-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ce0df8694f9a404ea9292cec4d688550
I20260924 15:23:28.975077 11184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:28.976152 11290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:28.976259 11289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.978768 11184 server_base.cc:1034] running on GCE node
W20260924 15:23:28.978834 11292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:28.978958 11184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:28.979123 11184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:28.980249 11184 hybrid_clock.cc:648] HybridClock initialized: now 1790263408980242 us; error 21 us; skew 500 ppm
Sep 24 15:23:28 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263408, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:28.981518 11184 init.cc:377] Logged in from keytab as oryx/127.4.78.124@KRBTEST.COM (short username oryx)
I20260924 15:23:28.981793 11184 webserver.cc:460] Webserver started at http://127.4.78.124:46047/ using document root <none> and password file <none>
I20260924 15:23:28.981940 11184 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:28.981988 11184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:28.982590 11184 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:28.983100 11300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:28.983222 11184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:28.983268 11184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
uuid: "ce0df8694f9a404ea9292cec4d688550"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "b3cb58b9100e16f4a674ae052834a74c"
server_key_iv: "2911d07db111ef79462ccf635fa3df9a"
server_key_version: "encryptionkey@0"
I20260924 15:23:28.983330 11184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:29.004568 11184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:29.006155 11184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:29.009356 11184 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.124:46113
I20260924 15:23:29.009538 11352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.124:46113 every 8 connection(s)
I20260924 15:23:29.009727 11184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
I20260924 15:23:29.010411 11353 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:23:29.010840 11353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Bootstrap starting.
I20260924 15:23:29.013419 11353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:23:29.013643 11353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Bootstrap complete.
I20260924 15:23:29.013885 11353 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } }
I20260924 15:23:29.013964 11353 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ce0df8694f9a404ea9292cec4d688550, State: Initialized, Role: LEARNER
I20260924 15:23:29.013991 11353 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } }
I20260924 15:23:29.014168 11353 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:29.014178 11356 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:29.014317 11356 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:23:29.015590 11367 catalog_manager.cc:1259] Loaded cluster ID: e330cec1d2cb47888aa35da6e67637f9
I20260924 15:23:29.015653 11367 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: loading cluster ID for follower catalog manager: success
I20260924 15:23:29.016579 11367 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: acquiring CA information for follower catalog manager: success
I20260924 15:23:29.016840 11367 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:23:29.053472 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.047171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.4.78.124:46113)
0924 15:23:29.047301 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.047303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.047318 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:23:29.047341 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.047342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.047351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.047409 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.048110 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.048729 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.049702 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.049960 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.050012 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.051203 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.051204 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.051204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.051210 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.052514 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.052822 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.052823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.052823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.052871 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.053026 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.053027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.053027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.053131 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.053201 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.053295 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.053354 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260924 15:23:29.053987 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 10588
I20260924 15:23:29.058815 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:37007
--local_ip_for_outbound_sockets=127.4.78.65
--tserver_master_addrs=127.4.78.126:39321
--webserver_port=42723
--webserver_interface=127.4.78.65
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab
--principal=oryx/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925,127.4.78.124:46113 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:29.141548 11369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:29.141738 11369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:29.141762 11369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:29.142874 11369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:29.142908 11369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:29.142935 11369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:29.144332 11369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab
--principal=oryx/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:37007
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=42723
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925,127.4.78.124:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:29.144512 11369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:29.144675 11369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:29.149626 11375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:29.149633 11374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:29.149857 11377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:29.149837 11369 server_base.cc:1034] running on GCE node
I20260924 15:23:29.150106 11369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:29.150352 11369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:29.151561 11369 hybrid_clock.cc:648] HybridClock initialized: now 1790263409151531 us; error 35 us; skew 500 ppm
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:29.153950 11369 init.cc:377] Logged in from keytab as oryx/127.4.78.65@KRBTEST.COM (short username oryx)
I20260924 15:23:29.154368 11369 webserver.cc:460] Webserver started at http://127.4.78.65:42723/ using document root <none> and password file <none>
I20260924 15:23:29.154542 11369 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:29.154593 11369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:29.155817 11369 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:29.156379 11384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.156560 11369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:29.156646 11369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "c437b618abc1229fc875e8753f925364"
server_key_iv: "3a4ea3a16c1fb56c9e7e74c1b73d3262"
server_key_version: "encryptionkey@0"
I20260924 15:23:29.156953 11369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:29.179267 11369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:29.181656 11369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:29.181775 11369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:29.182030 11369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:29.182303 11369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:29.182341 11369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.182365 11369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:29.182379 11369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.188375 11369 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:37007
I20260924 15:23:29.188455 11498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:37007 every 8 connection(s)
I20260924 15:23:29.188805 11369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:29.193475 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 11369
I20260924 15:23:29.193626 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 10724
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.125@KRBTEST.COM
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.124@KRBTEST.COM
I20260924 15:23:29.202428 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.189499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:50245 (local address 127.4.78.126:39321)
0924 15:23:29.189548 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.189550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.190047 (+ 497us) server_negotiation.cc:407] Connection header received
0924 15:23:29.191043 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.191045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.191060 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.191097 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.192022 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.192886 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.194043 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.194349 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.194391 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.197966 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.197968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.197968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.197975 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.200642 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.201188 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.201189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.201190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.201261 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.201565 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.201566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.201567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.201714 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.201777 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.202228 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.202323 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:23:29.202822 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.189784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:48455 (local address 127.4.78.124:46113)
0924 15:23:29.189855 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.189857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.189868 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:23:29.191046 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.191047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.191063 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.191097 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.192024 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.192892 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.195173 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.195536 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.195589 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.199022 (+ 3433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.199023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.199023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.199031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.201437 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.201876 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.201877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.201878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.201938 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.202148 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.202149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.202150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.202283 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.202340 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.202604 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.202721 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:23:29.203222 11505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.189518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:50245)
0924 15:23:29.189975 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.189986 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.190902 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.191182 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.191183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.191675 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.191898 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.191905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.192993 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.192994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.193950 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.193952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.194468 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.194469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.194532 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.195203 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.195209 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.197855 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.200730 (+ 2875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.200737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.200739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.201083 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.201335 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.201336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.201336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.201452 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.201851 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.201857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.202043 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.202661 (+ 618us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.202840 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":79,"thread_start_us":100,"threads_started":1}
I20260924 15:23:29.203222 11503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.189518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.4.78.65:48455)
0924 15:23:29.189706 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.189742 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.190900 (+ 1158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.191173 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.191174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.191691 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.191898 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.191905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.192993 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.192994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.194608 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.194610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.195687 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.195688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.195758 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.196226 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.196230 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.198907 (+ 2677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.201520 (+ 2613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.201521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.201522 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.201755 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.202009 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.202010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.202011 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.202068 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.202377 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.202378 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.202419 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.202951 (+ 532us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.203046 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":111,"thread_start_us":97,"threads_started":1}
I20260924 15:23:29.203460 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.189939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:60609 (local address 127.4.78.125:37925)
0924 15:23:29.189985 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.189987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.190078 (+ 91us) server_negotiation.cc:407] Connection header received
0924 15:23:29.191008 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.191009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.191022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.191075 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.191990 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.192575 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.193910 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.194230 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.194284 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.198517 (+ 4233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.198518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.198525 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.198533 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.202240 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.202634 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.202635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.202636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.202688 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.202877 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.202878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.202879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.203025 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.203069 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.203243 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.203345 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:23:29.203507 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:46657
--local_ip_for_outbound_sockets=127.4.78.66
--tserver_master_addrs=127.4.78.126:39321
--webserver_port=35133
--webserver_interface=127.4.78.66
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab
--principal=oryx/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925,127.4.78.124:46113 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:23:29.203814 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.189854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.4.78.65:60609)
0924 15:23:29.190022 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.190032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.190912 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.191153 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.191156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.191659 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.191898 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.191903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.192691 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.192692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.193788 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.193791 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.194376 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.194377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.194518 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.195155 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.195165 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.198419 (+ 3254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.202336 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.202337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.202338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.202557 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.202753 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.202754 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.202754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.202812 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.203101 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.203102 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.203142 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.203541 (+ 399us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.203666 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260924 15:23:29.204279 11500 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39321
I20260924 15:23:29.204398 11500 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.204587 11500 heartbeater.cc:511] Master 127.4.78.126:39321 requested a full tablet report, sending...
I20260924 15:23:29.205057 10524 ts_manager.cc:194] Re-registered known tserver with Master: 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007)
I20260924 15:23:29.205228 11501 heartbeater.cc:347] Connected to a master server at 127.4.78.125:37925
I20260924 15:23:29.205210 11499 heartbeater.cc:347] Connected to a master server at 127.4.78.124:46113
I20260924 15:23:29.205309 11501 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.205322 11499 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.205380 11501 heartbeater.cc:511] Master 127.4.78.125:37925 requested a full tablet report, sending...
I20260924 15:23:29.205389 11499 heartbeater.cc:511] Master 127.4.78.124:46113 requested a full tablet report, sending...
I20260924 15:23:29.205597 10524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:50245
I20260924 15:23:29.205776 11129 ts_manager.cc:194] Registered new tserver with Master: 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007)
I20260924 15:23:29.206034 11317 ts_manager.cc:194] Registered new tserver with Master: 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007)
W20260924 15:23:29.288229 11506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:29.288416 11506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:29.288439 11506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:29.289601 11506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:29.289642 11506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:29.289697 11506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:29.291276 11506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab
--principal=oryx/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:46657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=35133
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925,127.4.78.124:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:29.291507 11506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:29.291762 11506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:29.296546 11514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:29.296552 11512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:29.296728 11506 server_base.cc:1034] running on GCE node
W20260924 15:23:29.296557 11511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:29.296979 11506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:29.297267 11506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:29.298435 11506 hybrid_clock.cc:648] HybridClock initialized: now 1790263409298423 us; error 28 us; skew 500 ppm
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:29.300814 11506 init.cc:377] Logged in from keytab as oryx/127.4.78.66@KRBTEST.COM (short username oryx)
I20260924 15:23:29.301276 11506 webserver.cc:460] Webserver started at http://127.4.78.66:35133/ using document root <none> and password file <none>
I20260924 15:23:29.301517 11506 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:29.301599 11506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:29.302939 11506 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:29.303495 11521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.303663 11506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:29.303762 11506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "a2d1ee0740124cdbbb87a5f37d1854ea"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "0174fb320cc0c19916a0df7318286724"
server_key_iv: "3522e4486a2b042cd483809e01b4fea4"
server_key_version: "encryptionkey@0"
I20260924 15:23:29.304092 11506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:29.325611 11506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:29.327508 11506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:29.327669 11506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:29.327903 11506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:29.328203 11506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:29.328255 11506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.328310 11506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:29.328336 11506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.334134 11506 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:46657
I20260924 15:23:29.334177 11634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:46657 every 8 connection(s)
I20260924 15:23:29.334508 11506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:29.338829 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 11506
I20260924 15:23:29.338981 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 10860
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.124@KRBTEST.COM
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.125@KRBTEST.COM
I20260924 15:23:29.346323 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.335299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:51667 (local address 127.4.78.126:39321)
0924 15:23:29.335355 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.335357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.335672 (+ 315us) server_negotiation.cc:407] Connection header received
0924 15:23:29.336726 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.336728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.336740 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.336780 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.337551 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.338330 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.339530 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.339948 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.339986 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.343008 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.343009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.343010 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.343017 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.344596 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.345105 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.345106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.345106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.345165 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.345492 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.345493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.345494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.345635 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.345681 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.346123 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.346208 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 15:23:29.347114 11640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.335231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.66:51667)
0924 15:23:29.335549 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.335593 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.336604 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.336856 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.336859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.337271 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.337443 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.337449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.338431 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.338432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.339424 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.339427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.340060 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.340061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.340119 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.340815 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.340841 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.342907 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.344683 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.344692 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.344694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.345005 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.345246 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.345247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.345248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.345400 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.345732 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.345738 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.345932 (+ 194us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.346617 (+ 685us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.346862 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":108,"thread_start_us":212,"threads_started":1}
I20260924 15:23:29.347826 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.335725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:52913 (local address 127.4.78.125:37925)
0924 15:23:29.335767 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.335769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.335815 (+ 46us) server_negotiation.cc:407] Connection header received
0924 15:23:29.336723 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.336724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.336739 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.336788 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.337544 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.338121 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.339454 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.339865 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.339912 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.344669 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.344671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.344671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.344688 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.346454 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.346980 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.346981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.346981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.347039 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.347227 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.347228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.347228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.347359 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.347400 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.347591 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.347687 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:23:29.347882 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.335226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:47671 (local address 127.4.78.124:46113)
0924 15:23:29.335314 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.335316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.335671 (+ 355us) server_negotiation.cc:407] Connection header received
0924 15:23:29.336725 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.336726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.336740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.336772 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.337551 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.338335 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.339529 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.339810 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.339850 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.344339 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.344340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.344340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.344349 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.346600 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.347077 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.347078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.347079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.347141 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.347354 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.347354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.347355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.347475 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.347515 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.347697 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.347782 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 15:23:29.348278 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.335231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.4.78.66:47671)
0924 15:23:29.335549 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.335593 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.336604 (+ 1011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.336856 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.336859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.337279 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.337442 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.337449 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.338431 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.338432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.339424 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.339427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.339936 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.339937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.340076 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.340876 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.340880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.344241 (+ 3361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.346689 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.346690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.346691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.346989 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.347227 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.347228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.347228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.347291 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.347560 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.347560 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.347600 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.347955 (+ 355us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.348127 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":113,"thread_start_us":210,"threads_started":1}
I20260924 15:23:29.348855 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:39383
--local_ip_for_outbound_sockets=127.4.78.67
--tserver_master_addrs=127.4.78.126:39321
--webserver_port=43095
--webserver_interface=127.4.78.67
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab
--principal=oryx/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925,127.4.78.124:46113 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:23:29.348434 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.335648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.4.78.66:52913)
0924 15:23:29.335763 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.335773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.336617 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.336855 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.336859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.337250 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.337442 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.337447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.338226 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.338227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.339349 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.339352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.339992 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.339993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.340079 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.341007 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.341012 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.344575 (+ 3563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.346544 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.346545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.346545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.346896 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.347105 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.347106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.347106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.347163 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.347444 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.347445 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.347487 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.347912 (+ 425us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.348046 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260924 15:23:29.349216 11636 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39321
I20260924 15:23:29.349320 11636 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.349416 11637 heartbeater.cc:347] Connected to a master server at 127.4.78.125:37925
I20260924 15:23:29.349452 11637 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.349515 11636 heartbeater.cc:511] Master 127.4.78.126:39321 requested a full tablet report, sending...
I20260924 15:23:29.349514 11637 heartbeater.cc:511] Master 127.4.78.125:37925 requested a full tablet report, sending...
I20260924 15:23:29.349216 11635 heartbeater.cc:347] Connected to a master server at 127.4.78.124:46113
I20260924 15:23:29.349797 11635 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.349879 11635 heartbeater.cc:511] Master 127.4.78.124:46113 requested a full tablet report, sending...
I20260924 15:23:29.350030 11129 ts_manager.cc:194] Registered new tserver with Master: a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657)
I20260924 15:23:29.350292 10524 ts_manager.cc:194] Re-registered known tserver with Master: a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657)
I20260924 15:23:29.350441 11317 ts_manager.cc:194] Registered new tserver with Master: a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657)
I20260924 15:23:29.350734 10524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:51667
W20260924 15:23:29.430166 11642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:29.430365 11642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:29.430388 11642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:29.431463 11642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:29.431493 11642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:29.431519 11642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:29.432852 11642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab
--principal=oryx/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:39383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=43095
--tserver_master_addrs=127.4.78.126:39321,127.4.78.125:37925,127.4.78.124:46113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:29.433019 11642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:29.433187 11642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:29.438014 11647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:29.438023 11650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:29.438026 11648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:29.438254 11642 server_base.cc:1034] running on GCE node
I20260924 15:23:29.438453 11642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:29.438724 11642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:29.439882 11642 hybrid_clock.cc:648] HybridClock initialized: now 1790263409439866 us; error 22 us; skew 500 ppm
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:29.442239 11642 init.cc:377] Logged in from keytab as oryx/127.4.78.67@KRBTEST.COM (short username oryx)
I20260924 15:23:29.442656 11642 webserver.cc:460] Webserver started at http://127.4.78.67:43095/ using document root <none> and password file <none>
I20260924 15:23:29.442886 11642 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:29.442961 11642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:29.444288 11642 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:29.444831 11657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.445005 11642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.445072 11642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "fee2270f7c7647f89b868163f951feb2"
format_stamp: "Formatted at 2026-09-24 15:23:28 on dist-test-slave-l766"
server_key: "37573384f06d57b440a071ac6299b046"
server_key_iv: "b20c7b9927fe2c7a3f1d62cd82fe62f6"
server_key_version: "encryptionkey@0"
I20260924 15:23:29.445505 11642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:29.456686 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.453616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.0.0.1:53860)
0924 15:23:29.453728 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.453739 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.453853 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.454064 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.454065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.454067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:29.454119 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.454121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.454995 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.454995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.455763 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.455765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.456483 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.456484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.456538 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.456542 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.456587 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.456625 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":39,"threads_started":1}
I20260924 15:23:29.456692 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.453694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53860 (local address 127.4.78.124:46113)
0924 15:23:29.453764 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.453766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.453795 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:23:29.453928 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.453929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.453952 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.454018 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:29.454184 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.454923 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.455841 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.456403 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.456470 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.456533 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.456590 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.456618 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:23:29.457228 11327 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 15:23:29.457408 11356 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b363fdc18bf4c75b90080f5bb7205db. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:29.457502 11356 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:23:29.480351 11642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:29.482158 11642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:29.482270 11642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:29.482460 11642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:29.482860 11642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:29.482918 11642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.482959 11642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:29.482983 11642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:29.488967 11642 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:39383
I20260924 15:23:29.489012 11772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:39383 every 8 connection(s)
I20260924 15:23:29.489354 11642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:29.494251 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 11642
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.125@KRBTEST.COM
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
Sep 24 15:23:29 dist-test-slave-l766 krb5kdc[10479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263409, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.124@KRBTEST.COM
I20260924 15:23:29.502977 11181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.490109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:43043 (local address 127.4.78.125:37925)
0924 15:23:29.490148 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.490150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.490375 (+ 225us) server_negotiation.cc:407] Connection header received
0924 15:23:29.491000 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.491002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.491015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.491048 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.492014 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.492639 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.494049 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.494352 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.494411 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.498203 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.498205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.498205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.498214 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.499924 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.500431 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.500432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.500432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.500499 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.500757 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.500760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.500760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.500909 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.500977 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.502786 (+ 1809us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.502871 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:23:29.503551 11777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.490065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.4.78.67:43043)
0924 15:23:29.490258 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.490291 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.490875 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.491131 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.491135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.491644 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.491893 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.491899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.492754 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.492754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.493972 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.493973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.494458 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.494459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.494568 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.495496 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.495501 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.498109 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.500040 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.500045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.500047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.500353 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.500558 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.500558 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.500559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.500688 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.502448 (+ 1760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.502455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.502621 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.503111 (+ 490us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.503306 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":89,"spinlock_wait_cycles":384,"thread_start_us":93,"threads_started":1}
I20260924 15:23:29.504144 11775 heartbeater.cc:347] Connected to a master server at 127.4.78.125:37925
I20260924 15:23:29.504289 11775 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.504853 11775 heartbeater.cc:511] Master 127.4.78.125:37925 requested a full tablet report, sending...
I20260924 15:23:29.505285 11129 ts_manager.cc:194] Registered new tserver with Master: fee2270f7c7647f89b868163f951feb2 (127.4.78.67:39383)
I20260924 15:23:29.505360 11788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.495741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.4.78.124:46113)
0924 15:23:29.495997 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.495999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.496924 (+ 925us) server_negotiation.cc:407] Connection header received
0924 15:23:29.496951 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.496951 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.496963 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.497032 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.497758 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.498523 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.499617 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.499924 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.499965 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.501727 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.501728 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.501728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.501735 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.503962 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.504362 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.504363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.504364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.504420 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.504627 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.504628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.504628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.504753 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.504846 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.505169 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.505246 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":61,"threads_started":1}
I20260924 15:23:29.506085 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.495577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45484 (local address 127.4.78.125:37925)
0924 15:23:29.495769 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.495771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.496169 (+ 398us) server_negotiation.cc:407] Connection header received
0924 15:23:29.496193 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.496194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.496205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.496268 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.498080 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.498802 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.499913 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.500206 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.500250 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.502440 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.502442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.502442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.502449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.504115 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.504507 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.504509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.504509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.504571 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.505097 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.505098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.505098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.505239 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.505312 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.505875 (+ 563us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.505958 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":47,"threads_started":1}
I20260924 15:23:29.507040 10575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.490050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:53985 (local address 127.4.78.126:39321)
0924 15:23:29.490098 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.490100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.490525 (+ 425us) server_negotiation.cc:407] Connection header received
0924 15:23:29.491000 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.491002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.491015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.491048 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.492014 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.492801 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.494074 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.494376 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.494444 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.499981 (+ 5537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.499982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.499983 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.499991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.503159 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.506033 (+ 2874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.506035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.506035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.506096 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.506326 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.506327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.506327 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.506455 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.506508 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.506794 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.506922 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:29.507227 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.490066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:53985)
0924 15:23:29.490462 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.490472 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.490876 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.491131 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.491135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.491644 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.491893 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.491899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.492873 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.492874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.493965 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.493967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.494507 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.494508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.494591 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.495309 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.495320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.498070 (+ 2750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.503577 (+ 5507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.503578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.503579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.503803 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.506174 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.506175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.506175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.506235 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.506627 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.506628 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.506671 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.506834 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.506936 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":50,"thread_start_us":93,"threads_started":1}
I20260924 15:23:29.507793 11774 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39321
I20260924 15:23:29.507858 11774 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.507892 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.490430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:42315 (local address 127.4.78.124:46113)
0924 15:23:29.490475 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.490477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.490567 (+ 90us) server_negotiation.cc:407] Connection header received
0924 15:23:29.490978 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.490979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.490991 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.491037 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.492038 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.492790 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.493922 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.494220 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.494291 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.503897 (+ 9606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.503898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.503899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.503907 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.506221 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.506603 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.506604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.506605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.506657 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.507329 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.507330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.507330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.507466 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.507499 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.507730 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.507802 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:29.507942 11774 heartbeater.cc:511] Master 127.4.78.126:39321 requested a full tablet report, sending...
I20260924 15:23:29.508023 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.490352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.4.78.67:42315)
0924 15:23:29.490513 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.490524 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.490888 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.491512 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.491513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.491665 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:29.491892 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.491897 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.492890 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.492890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.493782 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.493785 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.494427 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.494428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.494569 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.495602 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:29.495605 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:29.499863 (+ 4258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:29.506299 (+ 6436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.506300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.506300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.506505 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.506744 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:29.506745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:29.506745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:29.507240 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:29.507576 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:29.507576 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:29.507617 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.507704 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.507848 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":85,"thread_start_us":72,"threads_started":1}
I20260924 15:23:29.508184 10524 ts_manager.cc:194] Re-registered known tserver with Master: fee2270f7c7647f89b868163f951feb2 (127.4.78.67:39383)
I20260924 15:23:29.508358 11773 heartbeater.cc:347] Connected to a master server at 127.4.78.124:46113
I20260924 15:23:29.508409 11773 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:29.508477 11773 heartbeater.cc:511] Master 127.4.78.124:46113 requested a full tablet report, sending...
I20260924 15:23:29.508672 10524 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:53985
I20260924 15:23:29.508746 11317 ts_manager.cc:194] Registered new tserver with Master: fee2270f7c7647f89b868163f951feb2 (127.4.78.67:39383)
I20260924 15:23:29.509680 11787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.495661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.4.78.126:39321)
0924 15:23:29.495819 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.495820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.496006 (+ 186us) server_negotiation.cc:407] Connection header received
0924 15:23:29.496032 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.496033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.496046 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.496113 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.497045 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.497720 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.501536 (+ 3816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.501943 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.502002 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.504994 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.504995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.504995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.505002 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.507081 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.507517 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.507518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.507518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.507580 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.508800 (+ 1220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.508801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.508801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.508921 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.509395 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.509502 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.509571 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":56,"threads_started":1}
I20260924 15:23:29.511207 10524 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:29.512806 10524 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:29.513441 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: NON_VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: true } } }
I20260924 15:23:29.513543 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:29.514557 11662 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: attempting to promote NON_VOTER ce0df8694f9a404ea9292cec4d688550 to VOTER
I20260924 15:23:29.514688 11662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } }
I20260924 15:23:29.515323 11327 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 LEARNER]: Refusing update from remote peer 4b363fdc18bf4c75b90080f5bb7205db: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 15:23:29.515326 11139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Refusing update from remote peer 4b363fdc18bf4c75b90080f5bb7205db: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 15:23:29.515600 11792 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 15:23:29.515714 11791 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 15:23:29.516428 11662 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ce0df8694f9a404ea9292cec4d688550 (127.4.78.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:29.516584 11327 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ce0df8694f9a404ea9292cec4d688550 (127.4.78.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:29.517267 11792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:29.517362 11792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:29.517324 11139 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, ce0df8694f9a404ea9292cec4d688550 (127.4.78.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:29.517409 11792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:29.517459 11792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:29.517278 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:29.517565 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:29.517511 11356 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:29.517591 11356 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:23:29.518302 11800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b363fdc18bf4c75b90080f5bb7205db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:29.518396 11800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:23:29.523284 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.519054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:39383 (local address 127.0.0.1:33260)
0924 15:23:29.519096 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.519106 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.519143 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.519792 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.519793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.519794 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:29.519819 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.519820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.520954 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.520955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.522183 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.522186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.523075 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.523076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.523126 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.523129 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.523156 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.523189 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":41}
I20260924 15:23:29.523384 11802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.519163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:46657 (local address 127.0.0.1:37424)
0924 15:23:29.519298 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.519310 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.519360 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.519884 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.519884 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.519886 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:29.519916 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.519917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.521147 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.521147 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.522282 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.522284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.523116 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.523117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.523179 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.523185 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.523244 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.523283 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260924 15:23:29.523514 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.519427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.4.78.66:46657)
0924 15:23:29.519617 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.519622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.519631 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:29.519673 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.519675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.519689 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.519795 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:29.519989 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.521072 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.522347 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.522982 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.523112 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.523190 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.523258 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.523295 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
I20260924 15:23:29.523625 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.519441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33260 (local address 127.4.78.67:39383)
0924 15:23:29.519626 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.519630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.519639 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:29.519672 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.519675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.519691 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.519797 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:29.519871 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.520879 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.522311 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.522974 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.523247 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.523340 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.523367 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.523541 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260924 15:23:29.523729 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.519241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.4.78.65:37007)
0924 15:23:29.519435 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.519440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.519448 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:29.519494 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.519495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.519508 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.519612 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:29.519733 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.520924 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.522069 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.522820 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.523495 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.523577 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.523602 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.523646 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260924 15:23:29.523204 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.519137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:37007 (local address 127.0.0.1:48344)
0924 15:23:29.519246 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.519256 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.519294 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.519612 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.519613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.519616 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:29.519665 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.519667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.521028 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.521029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.521984 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.521986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.522951 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.522952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.523018 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.523024 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.523082 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.523122 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":76,"threads_started":1}
I20260924 15:23:29.524324 11707 tablet_service.cc:1467] Processing CreateTablet for tablet d2a6a3b5329747a890ec38274abfafea (DEFAULT_TABLE table=test-table [id=ace2def5db29432280f0bf68792d33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:29.524325 11569 tablet_service.cc:1467] Processing CreateTablet for tablet d2a6a3b5329747a890ec38274abfafea (DEFAULT_TABLE table=test-table [id=ace2def5db29432280f0bf68792d33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:29.524503 11433 tablet_service.cc:1467] Processing CreateTablet for tablet d2a6a3b5329747a890ec38274abfafea (DEFAULT_TABLE table=test-table [id=ace2def5db29432280f0bf68792d33da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:29.524655 11569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a6a3b5329747a890ec38274abfafea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:29.524673 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a6a3b5329747a890ec38274abfafea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:29.524827 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2a6a3b5329747a890ec38274abfafea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:29.527479 11806 tablet_bootstrap.cc:492] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Bootstrap starting.
I20260924 15:23:29.528039 11807 tablet_bootstrap.cc:492] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Bootstrap starting.
I20260924 15:23:29.528187 11806 tablet_bootstrap.cc:654] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:29.528268 11808 tablet_bootstrap.cc:492] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Bootstrap starting.
I20260924 15:23:29.528551 11806 log.cc:824] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:29.528707 11807 tablet_bootstrap.cc:654] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:29.528924 11808 tablet_bootstrap.cc:654] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:29.529059 11807 log.cc:824] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:29.529249 11808 log.cc:824] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:29.529583 11806 tablet_bootstrap.cc:492] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: No bootstrap required, opened a new log
I20260924 15:23:29.529639 11806 ts_tablet_manager.cc:1397] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:29.530063 11808 tablet_bootstrap.cc:492] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: No bootstrap required, opened a new log
I20260924 15:23:29.530133 11808 ts_tablet_manager.cc:1397] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:29.530331 11807 tablet_bootstrap.cc:492] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: No bootstrap required, opened a new log
I20260924 15:23:29.530411 11807 ts_tablet_manager.cc:1397] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:29.531680 11806 raft_consensus.cc:348] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.531873 11806 raft_consensus.cc:374] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:29.531910 11806 raft_consensus.cc:729] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee2270f7c7647f89b868163f951feb2, State: Initialized, Role: FOLLOWER
I20260924 15:23:29.532032 11806 consensus_queue.cc:260] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.532307 11806 ts_tablet_manager.cc:1428] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:23:29.532302 11807 raft_consensus.cc:348] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.532435 11774 heartbeater.cc:503] Master 127.4.78.126:39321 was elected leader, sending a full tablet report...
I20260924 15:23:29.532462 11807 raft_consensus.cc:374] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:29.532507 11807 raft_consensus.cc:729] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70359e1c720b4ac98d0af0a0c1f6e9a5, State: Initialized, Role: FOLLOWER
I20260924 15:23:29.532619 11807 consensus_queue.cc:260] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.532886 11807 ts_tablet_manager.cc:1428] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:23:29.532884 11808 raft_consensus.cc:348] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.533166 11500 heartbeater.cc:503] Master 127.4.78.126:39321 was elected leader, sending a full tablet report...
I20260924 15:23:29.533030 11808 raft_consensus.cc:374] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:29.533218 11808 raft_consensus.cc:729] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d1ee0740124cdbbb87a5f37d1854ea, State: Initialized, Role: FOLLOWER
I20260924 15:23:29.533342 11808 consensus_queue.cc:260] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.533596 11808 ts_tablet_manager.cc:1428] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:23:29.533708 11636 heartbeater.cc:503] Master 127.4.78.126:39321 was elected leader, sending a full tablet report...
W20260924 15:23:29.585693 11638 tablet.cc:2369] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:23:29.689957 11502 tablet.cc:2369] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:29.698793 11812 raft_consensus.cc:482] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:29.698951 11812 raft_consensus.cc:504] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.699311 11812 leader_election.cc:288] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657), 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007)
I20260924 15:23:29.703329 11803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.699589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:39415 (local address 127.4.78.65:37007)
0924 15:23:29.699632 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.699634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.699643 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:29.699700 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.699701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.699713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.699790 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:29.700034 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.700800 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.702381 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.702922 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.703001 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.703013 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.703234 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.703271 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:23:29.703329 11804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.699572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:51109 (local address 127.4.78.66:46657)
0924 15:23:29.699621 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.699623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.699635 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:23:29.699701 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.699702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.699713 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.699783 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:29.700032 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.700797 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.702379 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.702921 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.703001 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.703013 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.703211 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.703252 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:23:29.703346 11778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.699497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:46657 (local address 127.4.78.67:51109)
0924 15:23:29.699578 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.699591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.699633 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.699824 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.699826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.699829 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:29.699944 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.699946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.700885 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.700886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.702292 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.702295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.703028 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.703030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.703121 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.703128 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.703205 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.703250 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":79}
I20260924 15:23:29.703361 11779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.699497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:37007 (local address 127.4.78.67:39415)
0924 15:23:29.699578 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:29.699591 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:29.699633 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:29.699824 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:29.699826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:29.699831 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:29.699944 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.699946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.700886 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.700886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.702300 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:29.702301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.703028 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.703030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:29.703121 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.703128 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:23:29.703223 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 15:23:29.703254 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":172}
I20260924 15:23:29.703702 11589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a6a3b5329747a890ec38274abfafea" candidate_uuid: "fee2270f7c7647f89b868163f951feb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" is_pre_election: true
I20260924 15:23:29.703871 11589 raft_consensus.cc:2454] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee2270f7c7647f89b868163f951feb2 in term 0.
I20260924 15:23:29.703847 11453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a6a3b5329747a890ec38274abfafea" candidate_uuid: "fee2270f7c7647f89b868163f951feb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" is_pre_election: true
I20260924 15:23:29.703994 11453 raft_consensus.cc:2454] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee2270f7c7647f89b868163f951feb2 in term 0.
I20260924 15:23:29.704252 11658 leader_election.cc:302] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70359e1c720b4ac98d0af0a0c1f6e9a5, fee2270f7c7647f89b868163f951feb2; no voters:
I20260924 15:23:29.704420 11812 raft_consensus.cc:2781] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:29.704495 11812 raft_consensus.cc:482] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:29.704550 11812 raft_consensus.cc:3037] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:29.705250 11812 raft_consensus.cc:504] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.705408 11812 leader_election.cc:288] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [CANDIDATE]: Term 1 election: Requested vote from peers a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657), 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007)
I20260924 15:23:29.705521 11589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a6a3b5329747a890ec38274abfafea" candidate_uuid: "fee2270f7c7647f89b868163f951feb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea"
I20260924 15:23:29.705608 11453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2a6a3b5329747a890ec38274abfafea" candidate_uuid: "fee2270f7c7647f89b868163f951feb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5"
I20260924 15:23:29.705618 11589 raft_consensus.cc:3037] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:29.705704 11453 raft_consensus.cc:3037] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:29.706611 11453 raft_consensus.cc:2454] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee2270f7c7647f89b868163f951feb2 in term 1.
I20260924 15:23:29.706609 11589 raft_consensus.cc:2454] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee2270f7c7647f89b868163f951feb2 in term 1.
I20260924 15:23:29.706774 11658 leader_election.cc:302] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70359e1c720b4ac98d0af0a0c1f6e9a5, fee2270f7c7647f89b868163f951feb2; no voters:
I20260924 15:23:29.706887 11812 raft_consensus.cc:2781] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:29.707046 11812 raft_consensus.cc:686] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 1 LEADER]: Becoming Leader. State: Replica: fee2270f7c7647f89b868163f951feb2, State: Running, Role: LEADER
I20260924 15:23:29.707340 11812 consensus_queue.cc:237] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } }
I20260924 15:23:29.708202 10524 catalog_manager.cc:5495] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fee2270f7c7647f89b868163f951feb2 (127.4.78.67). New cstate: current_term: 1 leader_uuid: "fee2270f7c7647f89b868163f951feb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } health_report { overall_health: UNKNOWN } } }
W20260924 15:23:29.740191 11776 tablet.cc:2369] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:29.747656 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.744514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.4.78.67:39383)
0924 15:23:29.744588 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.744590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.744599 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:29.744653 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.744654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.744673 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.744736 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:29.744956 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.745948 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.746786 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.746996 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.747083 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.747262 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:29.747386 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:29.747482 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.747523 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.747550 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 15:23:29.749552 11589 raft_consensus.cc:1261] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 1 FOLLOWER]: Refusing update from remote peer fee2270f7c7647f89b868163f951feb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:29.749553 11453 raft_consensus.cc:1261] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 1 FOLLOWER]: Refusing update from remote peer fee2270f7c7647f89b868163f951feb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:29.749941 11812 consensus_queue.cc:1035] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:29.750092 11815 consensus_queue.cc:1035] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:29.753741 11819 mvcc.cc:205] Tried to move back new op lower bound from 7332918926331207680 to 7332918926161981440. Current Snapshot: MvccSnapshot[applied={T|T < 7332918926331207680 or (T in {7332918926331207680})}]
I20260924 15:23:29.754865 11820 mvcc.cc:205] Tried to move back new op lower bound from 7332918926331207680 to 7332918926161981440. Current Snapshot: MvccSnapshot[applied={T|T < 7332918926331207680}]
I20260924 15:23:29.755658 10524 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:29.755752 10524 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:29.757238 10524 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b363fdc18bf4c75b90080f5bb7205db: Sending DeleteTablet for 3 replicas of tablet d2a6a3b5329747a890ec38274abfafea
I20260924 15:23:29.757529 11707 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a6a3b5329747a890ec38274abfafea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:29 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:33260
I20260924 15:23:29.757599 11569 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a6a3b5329747a890ec38274abfafea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:29 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:37424
I20260924 15:23:29.757786 11829 tablet_replica.cc:330] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: stopping tablet replica
I20260924 15:23:29.757903 11829 raft_consensus.cc:2229] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:29.758009 11829 raft_consensus.cc:2258] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:29.758426 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 10492
I20260924 15:23:29.758452 11829 ts_tablet_manager.cc:1905] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:29.758452 11433 tablet_service.cc:1514] Processing DeleteTablet for tablet d2a6a3b5329747a890ec38274abfafea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:29 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:48344
I20260924 15:23:29.759754 11830 tablet_replica.cc:330] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: stopping tablet replica
I20260924 15:23:29.759866 11830 raft_consensus.cc:2229] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:29.759965 11830 raft_consensus.cc:2258] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:29.760118 11828 tablet_replica.cc:330] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: stopping tablet replica
I20260924 15:23:29.760196 11828 raft_consensus.cc:2229] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:29.760274 11828 raft_consensus.cc:2258] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:29.760672 11829 ts_tablet_manager.cc:1918] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:29.760735 11829 log.cc:1196] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/d2a6a3b5329747a890ec38274abfafea
I20260924 15:23:29.760744 11828 ts_tablet_manager.cc:1905] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:29.761003 11829 ts_tablet_manager.cc:1939] T d2a6a3b5329747a890ec38274abfafea P a2d1ee0740124cdbbb87a5f37d1854ea: Deleting consensus metadata
I20260924 15:23:29.762377 11830 ts_tablet_manager.cc:1905] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:29.763779 11830 ts_tablet_manager.cc:1918] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:29.763806 11828 ts_tablet_manager.cc:1918] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:29.763857 11830 log.cc:1196] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/d2a6a3b5329747a890ec38274abfafea
I20260924 15:23:29.763868 11828 log.cc:1196] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/d2a6a3b5329747a890ec38274abfafea
I20260924 15:23:29.764123 11830 ts_tablet_manager.cc:1939] T d2a6a3b5329747a890ec38274abfafea P 70359e1c720b4ac98d0af0a0c1f6e9a5: Deleting consensus metadata
I20260924 15:23:29.764132 11828 ts_tablet_manager.cc:1939] T d2a6a3b5329747a890ec38274abfafea P fee2270f7c7647f89b868163f951feb2: Deleting consensus metadata
W20260924 15:23:29.765089 11523 connection.cc:535] server connection from 127.0.0.1:37424 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 15:23:29.765089 11385 connection.cc:535] server connection from 127.0.0.1:48344 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 15:23:29.765259 11659 connection.cc:535] server connection from 127.0.0.1:33260 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:23:29.773480 11786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.767022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45498 (local address 127.4.78.125:37925)
0924 15:23:29.767061 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.767063 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.767070 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:29.767093 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.767094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.767116 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.767175 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.767732 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.768223 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.768943 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.769159 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.769236 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.770596 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.770597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.770598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.770605 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.772208 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.772610 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.772612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.772612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.772673 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.772889 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.772890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.772891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.773030 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.773133 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.773272 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.773359 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:29.774691 11368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:29.766941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53874 (local address 127.4.78.124:46113)
0924 15:23:29.766984 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:29.766986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:29.767135 (+ 149us) server_negotiation.cc:407] Connection header received
0924 15:23:29.767165 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:29.767166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:29.767176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:29.767228 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:29.767915 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.768672 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.769740 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:29.770067 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:29.770122 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:29.771709 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:29.771711 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:29.771711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:29.771719 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:29.773392 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.773934 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.773936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.773936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.773995 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:29.774198 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:29.774200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:29.774200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:29.774343 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:29.774445 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:29.774518 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 15:23:29.774593 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:23:30.754652 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:30.754395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:60401)
0924 15:23:30.754528 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:30.754606 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
W20260924 15:23:30.754896 11500 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:30.755736 11839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:30.755492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.66:55981)
0924 15:23:30.755625 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:30.755694 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":50,"threads_started":1}
W20260924 15:23:30.755972 11636 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:30.756151 11840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:30.755913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:41353)
0924 15:23:30.756048 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:30.756103 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":48,"threads_started":1}
W20260924 15:23:30.756319 11774 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:31.258208 11841 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b363fdc18bf4c75b90080f5bb7205db)
I20260924 15:23:31.258385 11841 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } }
I20260924 15:23:31.258395 11842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b363fdc18bf4c75b90080f5bb7205db)
I20260924 15:23:31.258544 11842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } }
I20260924 15:23:31.258694 11841 leader_election.cc:288] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321), 41f23ec6a5ce40208366634772d324b1 (127.4.78.125:37925)
I20260924 15:23:31.259012 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.258787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:41364)
0924 15:23:31.258921 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.258968 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":45,"threads_started":1}
I20260924 15:23:31.259141 11842 leader_election.cc:288] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321), ce0df8694f9a404ea9292cec4d688550 (127.4.78.124:46113)
W20260924 15:23:31.259140 11301 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:31.259485 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.258845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:41374)
0924 15:23:31.259020 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.259329 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":46,"threads_started":1}
W20260924 15:23:31.259611 11113 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:31.259706 11843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.259449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:41390)
0924 15:23:31.259486 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.259514 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 15:23:31.259825 11301 leader_election.cc:334] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321): Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:31.260015 11845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.259916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:41406)
0924 15:23:31.259946 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.259990 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 15:23:31.260080 11113 leader_election.cc:334] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321): Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:31.262056 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.258843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37925 (local address 127.0.0.1:45506)
0924 15:23:31.259003 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.259013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:31.259191 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:31.259546 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:31.259547 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:31.259550 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:31.259606 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:31.259608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:31.260433 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:31.260434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:31.261224 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:31.261226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:31.261768 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:31.261769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:31.261868 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:31.261875 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:23:31.261933 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:23:31.261980 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":45,"threads_started":1}
I20260924 15:23:31.262181 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.259031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:46113 (local address 127.0.0.1:53890)
0924 15:23:31.259287 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.259297 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:31.259351 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:31.259603 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:31.259603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:31.259606 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:31.259654 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:31.259657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:31.260587 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:31.260587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:31.261334 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:31.261335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:31.261887 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:31.261888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:31.261948 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:31.261951 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:31.262026 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:23:31.262073 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":32,"threads_started":1}
I20260924 15:23:31.262199 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.258891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45506 (local address 127.4.78.125:37925)
0924 15:23:31.258989 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:31.258991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:31.259103 (+ 112us) server_negotiation.cc:407] Connection header received
0924 15:23:31.259432 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:31.259433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:31.259442 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:31.259491 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:31.259696 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:31.260375 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:31.261306 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:31.261697 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:31.261749 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:31.261757 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:31.262037 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 15:23:31.262101 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":51,"threads_started":1}
I20260924 15:23:31.262437 11139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce0df8694f9a404ea9292cec4d688550" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41f23ec6a5ce40208366634772d324b1" is_pre_election: true
I20260924 15:23:31.262487 11849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.259269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.4.78.124:46113)
0924 15:23:31.259430 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:31.259433 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:31.259441 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:31.259466 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:31.259466 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:31.259481 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:31.259552 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:31.259735 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:31.260515 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:31.261412 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:31.261778 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:31.261868 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:31.261878 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:31.262054 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:23:31.262425 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
I20260924 15:23:31.262545 11139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce0df8694f9a404ea9292cec4d688550 in term 1.
I20260924 15:23:31.262701 11327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41f23ec6a5ce40208366634772d324b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ce0df8694f9a404ea9292cec4d688550" is_pre_election: true
I20260924 15:23:31.262709 11303 leader_election.cc:302] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41f23ec6a5ce40208366634772d324b1, ce0df8694f9a404ea9292cec4d688550; no voters: 4b363fdc18bf4c75b90080f5bb7205db
I20260924 15:23:31.262822 11327 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41f23ec6a5ce40208366634772d324b1 in term 1.
I20260924 15:23:31.262903 11841 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 15:23:31.262951 11841 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b363fdc18bf4c75b90080f5bb7205db)
I20260924 15:23:31.262974 11841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:23:31.263008 11115 leader_election.cc:302] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41f23ec6a5ce40208366634772d324b1, ce0df8694f9a404ea9292cec4d688550; no voters: 4b363fdc18bf4c75b90080f5bb7205db
I20260924 15:23:31.263151 11842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 15:23:31.263197 11842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b363fdc18bf4c75b90080f5bb7205db)
I20260924 15:23:31.263222 11842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:23:31.263877 11841 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } }
I20260924 15:23:31.264029 11841 leader_election.cc:288] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 2 election: Requested vote from peers 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321), 41f23ec6a5ce40208366634772d324b1 (127.4.78.125:37925)
I20260924 15:23:31.264055 11842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } }
I20260924 15:23:31.264168 11842 leader_election.cc:288] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [CANDIDATE]: Term 2 election: Requested vote from peers 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321), ce0df8694f9a404ea9292cec4d688550 (127.4.78.124:46113)
I20260924 15:23:31.264232 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.264130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:41416)
0924 15:23:31.264176 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.264200 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:23:31.264308 11327 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "41f23ec6a5ce40208366634772d324b1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ce0df8694f9a404ea9292cec4d688550"
I20260924 15:23:31.264400 11327 raft_consensus.cc:2379] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 41f23ec6a5ce40208366634772d324b1 in current term 2: Already voted for candidate ce0df8694f9a404ea9292cec4d688550 in this term.
I20260924 15:23:31.264415 11139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce0df8694f9a404ea9292cec4d688550" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41f23ec6a5ce40208366634772d324b1"
I20260924 15:23:31.264492 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.264279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:41420)
0924 15:23:31.264380 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.264406 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 15:23:31.264519 11139 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ce0df8694f9a404ea9292cec4d688550 in current term 2: Already voted for candidate 41f23ec6a5ce40208366634772d324b1 in this term.
I20260924 15:23:31.264765 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.264655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:41436)
0924 15:23:31.264707 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.264738 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 15:23:31.264778 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.264700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:41452)
0924 15:23:31.264734 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.264756 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 15:23:31.264827 11301 leader_election.cc:334] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321): Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
W20260924 15:23:31.264837 11113 leader_election.cc:334] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321): Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:31.264858 11301 leader_election.cc:302] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce0df8694f9a404ea9292cec4d688550; no voters: 41f23ec6a5ce40208366634772d324b1, 4b363fdc18bf4c75b90080f5bb7205db
I20260924 15:23:31.264899 11113 leader_election.cc:302] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 41f23ec6a5ce40208366634772d324b1; no voters: 4b363fdc18bf4c75b90080f5bb7205db, ce0df8694f9a404ea9292cec4d688550
I20260924 15:23:31.264950 11841 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 15:23:31.264986 11842 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 15:23:31.755582 11851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.755331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:45861)
0924 15:23:31.755476 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.755526 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
I20260924 15:23:31.756474 11852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.756249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.66:43687)
0924 15:23:31.756375 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.756406 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
I20260924 15:23:31.756764 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:31.756603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:35875)
0924 15:23:31.756711 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:31.756732 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":61,"threads_started":1}
I20260924 15:23:32.756376 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:32.756118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:37155)
0924 15:23:32.756258 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:32.756312 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260924 15:23:32.757020 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:32.756834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.66:35939)
0924 15:23:32.756943 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:32.756974 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260924 15:23:32.757314 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:32.757148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:41883)
0924 15:23:32.757257 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:32.757280 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
I20260924 15:23:33.062587 11857 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:33.062683 11857 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } }
I20260924 15:23:33.062847 11857 leader_election.cc:288] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321), 41f23ec6a5ce40208366634772d324b1 (127.4.78.125:37925)
I20260924 15:23:33.063115 11139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce0df8694f9a404ea9292cec4d688550" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41f23ec6a5ce40208366634772d324b1" is_pre_election: true
I20260924 15:23:33.063170 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.062985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:38050)
0924 15:23:33.063087 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.063131 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":60,"threads_started":1}
I20260924 15:23:33.063202 11139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce0df8694f9a404ea9292cec4d688550 in term 2.
I20260924 15:23:33.063413 11303 leader_election.cc:302] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41f23ec6a5ce40208366634772d324b1, ce0df8694f9a404ea9292cec4d688550; no voters:
I20260924 15:23:33.063539 11857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 15:23:33.063589 11857 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:33.063608 11857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 2 FOLLOWER]: Advancing to term 3
I20260924 15:23:33.063648 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.063533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:38060)
0924 15:23:33.063574 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.063622 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 15:23:33.063726 11301 leader_election.cc:334] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321): Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:33.064340 11857 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } }
I20260924 15:23:33.064477 11857 leader_election.cc:288] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 3 election: Requested vote from peers 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321), 41f23ec6a5ce40208366634772d324b1 (127.4.78.125:37925)
I20260924 15:23:33.064670 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.064545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:38068)
0924 15:23:33.064580 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.064630 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 15:23:33.064687 11139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce0df8694f9a404ea9292cec4d688550" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "41f23ec6a5ce40208366634772d324b1"
I20260924 15:23:33.064764 11139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 2 FOLLOWER]: Advancing to term 3
I20260924 15:23:33.065029 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.064905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:38074)
0924 15:23:33.064981 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.065009 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 15:23:33.065100 11301 leader_election.cc:334] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321): Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
I20260924 15:23:33.065616 11139 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce0df8694f9a404ea9292cec4d688550 in term 3.
I20260924 15:23:33.065788 11303 leader_election.cc:302] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 41f23ec6a5ce40208366634772d324b1, ce0df8694f9a404ea9292cec4d688550; no voters: 4b363fdc18bf4c75b90080f5bb7205db
I20260924 15:23:33.065937 11857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 15:23:33.066051 11857 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [term 3 LEADER]: Becoming Leader. State: Replica: ce0df8694f9a404ea9292cec4d688550, State: Running, Role: LEADER
I20260924 15:23:33.066147 11857 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } }
I20260924 15:23:33.066582 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0df8694f9a404ea9292cec4d688550. Latest consensus state: current_term: 3 leader_uuid: "ce0df8694f9a404ea9292cec4d688550" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:33.066686 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:33.066860 11862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:33.067216 11862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:33.067268 11862 catalog_manager.cc:1259] Loaded cluster ID: e330cec1d2cb47888aa35da6e67637f9
I20260924 15:23:33.067287 11862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:33.067430 11862 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:33.067481 11862 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Loaded TSK: 0
I20260924 15:23:33.067579 11862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:23:33.095930 11317 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:33.097642 11317 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:33.098719 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.098598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:38092)
0924 15:23:33.098654 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.098689 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:23:33.098757 11139 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [term 3 FOLLOWER]: Refusing update from remote peer ce0df8694f9a404ea9292cec4d688550: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 15:23:33.098978 11860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 15:23:33.099123 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.099025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.0.0.1:38108)
0924 15:23:33.099072 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.099103 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:23:33.099189 11301 consensus_peers.cc:487] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 -> Peer 4b363fdc18bf4c75b90080f5bb7205db (127.4.78.126:39321): Couldn't send request to peer 4b363fdc18bf4c75b90080f5bb7205db. Status: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 15:23:33.099639 11864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce0df8694f9a404ea9292cec4d688550. Latest consensus state: current_term: 3 leader_uuid: "ce0df8694f9a404ea9292cec4d688550" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:33.099747 11864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:23:33.100099 11863 mvcc.cc:205] Tried to move back new op lower bound from 7332918940050272256 to 7332918939919683584. Current Snapshot: MvccSnapshot[applied={T|T < 7332918940050272256}]
I20260924 15:23:33.100224 11860 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ce0df8694f9a404ea9292cec4d688550" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:33.100325 11860 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:33.100549 11864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ce0df8694f9a404ea9292cec4d688550" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4b363fdc18bf4c75b90080f5bb7205db" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39321 } } peers { permanent_uuid: "41f23ec6a5ce40208366634772d324b1" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37925 } attrs { promote: false } } peers { permanent_uuid: "ce0df8694f9a404ea9292cec4d688550" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 46113 } attrs { promote: false } } }
I20260924 15:23:33.100637 11864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41f23ec6a5ce40208366634772d324b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:23:33.100914 11865 mvcc.cc:205] Tried to move back new op lower bound from 7332918940050272256 to 7332918939919683584. Current Snapshot: MvccSnapshot[applied={T|T < 7332918940050272256 or (T in {7332918940050272256})}]
I20260924 15:23:33.105254 11858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.101974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:39383 (local address 127.0.0.1:48120)
0924 15:23:33.102047 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.102059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:33.102110 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:33.102317 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:33.102318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:33.102320 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:33.102374 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:33.102376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.103218 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.103219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:33.104271 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:33.104273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.105019 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.105020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:33.105082 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:33.105087 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:23:33.105140 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:23:33.105185 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 15:23:33.105278 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.102056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.4.78.67:39383)
0924 15:23:33.102167 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:33.102169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:33.102186 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:23:33.102213 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:33.102214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:33.102225 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:33.102308 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:33.102445 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.103132 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.104365 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.104938 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.105004 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:33.105015 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:33.105157 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 15:23:33.105206 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260924 15:23:33.105428 11873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.102250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34204 (local address 127.4.78.65:37007)
0924 15:23:33.102383 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:33.102385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:33.102433 (+ 48us) server_negotiation.cc:407] Connection header received
0924 15:23:33.102642 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:33.102643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:33.102654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:33.102713 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:33.102862 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.103570 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.104439 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.104970 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.105056 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:33.105067 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:33.105299 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 15:23:33.105344 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260924 15:23:33.105522 11707 tablet_service.cc:1467] Processing CreateTablet for tablet 01d7614fbda24fb883932e742d5518be (DEFAULT_TABLE table=test-table [id=a04203a4245d49bca45fe2106a87889e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:33.105568 11870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.102061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:37007 (local address 127.0.0.1:34204)
0924 15:23:33.102382 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.102392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:33.102445 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:33.102758 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:33.102759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:33.102761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:33.102791 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:33.102792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.103625 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.103627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:33.104347 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:33.104349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.105094 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.105095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:33.105155 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:33.105158 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:33.105200 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 15:23:33.105231 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":90,"threads_started":1}
I20260924 15:23:33.105662 11707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d7614fbda24fb883932e742d5518be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:33.105780 11433 tablet_service.cc:1467] Processing CreateTablet for tablet 01d7614fbda24fb883932e742d5518be (DEFAULT_TABLE table=test-table [id=a04203a4245d49bca45fe2106a87889e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:33.105921 11433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d7614fbda24fb883932e742d5518be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:33.106103 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.102056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:46657 (local address 127.0.0.1:35140)
0924 15:23:33.102253 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.102263 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:33.102443 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:33.102594 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:33.102594 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:33.102596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:33.102634 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:33.102636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.103555 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.103556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:33.104652 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:33.104654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.105883 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.105884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:33.105947 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:33.105951 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:33.106005 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:23:33.106039 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":30,"thread_start_us":76,"threads_started":1}
I20260924 15:23:33.106148 11872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.102250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.4.78.66:46657)
0924 15:23:33.102363 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:33.102365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:33.102374 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:33.102500 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:33.102501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:33.102511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:33.102581 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:33.102713 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.103473 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.104745 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.105295 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.105871 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:33.105888 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:33.106024 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 15:23:33.106074 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260924 15:23:33.107036 11874 tablet_bootstrap.cc:492] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: Bootstrap starting.
I20260924 15:23:33.107331 11569 tablet_service.cc:1467] Processing CreateTablet for tablet 01d7614fbda24fb883932e742d5518be (DEFAULT_TABLE table=test-table [id=a04203a4245d49bca45fe2106a87889e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:33.107414 11874 tablet_bootstrap.cc:654] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:33.107434 11569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01d7614fbda24fb883932e742d5518be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:33.107966 11874 tablet_bootstrap.cc:492] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: No bootstrap required, opened a new log
I20260924 15:23:33.108009 11874 ts_tablet_manager.cc:1397] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:33.108130 11874 raft_consensus.cc:348] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.108194 11874 raft_consensus.cc:374] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:33.108217 11874 raft_consensus.cc:729] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fee2270f7c7647f89b868163f951feb2, State: Initialized, Role: FOLLOWER
I20260924 15:23:33.108253 11874 consensus_queue.cc:260] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.108418 11874 ts_tablet_manager.cc:1428] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:33.108865 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.108730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:35735)
0924 15:23:33.108771 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.108828 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 15:23:33.109090 11878 tablet_bootstrap.cc:492] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: Bootstrap starting.
I20260924 15:23:33.109440 11878 tablet_bootstrap.cc:654] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:33.109757 11875 tablet_bootstrap.cc:492] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5: Bootstrap starting.
I20260924 15:23:33.110112 11875 tablet_bootstrap.cc:654] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:33.110211 11878 tablet_bootstrap.cc:492] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: No bootstrap required, opened a new log
I20260924 15:23:33.110288 11878 ts_tablet_manager.cc:1397] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:33.110435 11878 raft_consensus.cc:348] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.110505 11878 raft_consensus.cc:374] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:33.110520 11878 raft_consensus.cc:729] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d1ee0740124cdbbb87a5f37d1854ea, State: Initialized, Role: FOLLOWER
I20260924 15:23:33.110579 11878 consensus_queue.cc:260] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.110757 11878 ts_tablet_manager.cc:1428] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:33.110788 11875 tablet_bootstrap.cc:492] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5: No bootstrap required, opened a new log
I20260924 15:23:33.110827 11875 ts_tablet_manager.cc:1397] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:33.110973 11875 raft_consensus.cc:348] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.111049 11875 raft_consensus.cc:374] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:33.111058 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.110965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.66:34857)
0924 15:23:33.111000 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.111031 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 15:23:33.111078 11875 raft_consensus.cc:729] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70359e1c720b4ac98d0af0a0c1f6e9a5, State: Initialized, Role: FOLLOWER
I20260924 15:23:33.111215 11635 heartbeater.cc:503] Master 127.4.78.124:46113 was elected leader, sending a full tablet report...
I20260924 15:23:33.111202 11875 consensus_queue.cc:260] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.111389 11875 ts_tablet_manager.cc:1428] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:33.111720 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.111576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:53473)
0924 15:23:33.111650 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.111683 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 15:23:33.111981 11499 heartbeater.cc:503] Master 127.4.78.124:46113 was elected leader, sending a full tablet report...
I20260924 15:23:33.112033 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.111908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:34887)
0924 15:23:33.111944 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.111975 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 15:23:33.147225 11877 raft_consensus.cc:482] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:33.147330 11877 raft_consensus.cc:504] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.147528 11877 leader_election.cc:288] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007), a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657)
I20260924 15:23:33.147734 11589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d7614fbda24fb883932e742d5518be" candidate_uuid: "fee2270f7c7647f89b868163f951feb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" is_pre_election: true
I20260924 15:23:33.147814 11589 raft_consensus.cc:2454] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee2270f7c7647f89b868163f951feb2 in term 0.
I20260924 15:23:33.147754 11453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d7614fbda24fb883932e742d5518be" candidate_uuid: "fee2270f7c7647f89b868163f951feb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" is_pre_election: true
I20260924 15:23:33.147881 11453 raft_consensus.cc:2454] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fee2270f7c7647f89b868163f951feb2 in term 0.
I20260924 15:23:33.147979 11659 leader_election.cc:302] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2d1ee0740124cdbbb87a5f37d1854ea, fee2270f7c7647f89b868163f951feb2; no voters:
I20260924 15:23:33.148088 11877 raft_consensus.cc:2781] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:33.148135 11877 raft_consensus.cc:482] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:33.148149 11877 raft_consensus.cc:3037] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:33.148942 11877 raft_consensus.cc:504] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.149121 11877 leader_election.cc:288] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [CANDIDATE]: Term 1 election: Requested vote from peers 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007), a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657)
I20260924 15:23:33.149292 11453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d7614fbda24fb883932e742d5518be" candidate_uuid: "fee2270f7c7647f89b868163f951feb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5"
I20260924 15:23:33.149372 11453 raft_consensus.cc:3037] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:33.149333 11589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01d7614fbda24fb883932e742d5518be" candidate_uuid: "fee2270f7c7647f89b868163f951feb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea"
I20260924 15:23:33.149438 11589 raft_consensus.cc:3037] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:33.150024 11453 raft_consensus.cc:2454] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee2270f7c7647f89b868163f951feb2 in term 1.
I20260924 15:23:33.150166 11589 raft_consensus.cc:2454] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fee2270f7c7647f89b868163f951feb2 in term 1.
I20260924 15:23:33.150182 11658 leader_election.cc:302] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70359e1c720b4ac98d0af0a0c1f6e9a5, fee2270f7c7647f89b868163f951feb2; no voters:
I20260924 15:23:33.150261 11877 raft_consensus.cc:2781] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:33.150411 11877 raft_consensus.cc:686] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 1 LEADER]: Becoming Leader. State: Replica: fee2270f7c7647f89b868163f951feb2, State: Running, Role: LEADER
I20260924 15:23:33.150480 11877 consensus_queue.cc:237] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } }
I20260924 15:23:33.150532 11773 heartbeater.cc:503] Master 127.4.78.124:46113 was elected leader, sending a full tablet report...
I20260924 15:23:33.150936 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.150818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:48173)
0924 15:23:33.150870 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.150899 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:23:33.151141 11317 catalog_manager.cc:5495] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to fee2270f7c7647f89b868163f951feb2 (127.4.78.67). New cstate: current_term: 1 leader_uuid: "fee2270f7c7647f89b868163f951feb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fee2270f7c7647f89b868163f951feb2" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 39383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 } health_report { overall_health: UNKNOWN } } }
I20260924 15:23:33.163254 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.160944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48126 (local address 127.4.78.67:39383)
0924 15:23:33.160997 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:33.160999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:33.161008 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:33.161105 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:33.161106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:33.161118 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:33.161162 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:33.161319 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.161782 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.162578 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:33.162780 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:33.162851 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:33.163003 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:33.163039 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:33.163091 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:33.163155 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 15:23:33.163175 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:23:33.164062 11453 raft_consensus.cc:1261] T 01d7614fbda24fb883932e742d5518be P 70359e1c720b4ac98d0af0a0c1f6e9a5 [term 1 FOLLOWER]: Refusing update from remote peer fee2270f7c7647f89b868163f951feb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:33.164229 11589 raft_consensus.cc:1261] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 1 FOLLOWER]: Refusing update from remote peer fee2270f7c7647f89b868163f951feb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:33.164278 11883 consensus_queue.cc:1035] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70359e1c720b4ac98d0af0a0c1f6e9a5" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 37007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:33.164463 11883 consensus_queue.cc:1035] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d1ee0740124cdbbb87a5f37d1854ea" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 46657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:33.165287 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.165176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:40951)
0924 15:23:33.165220 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.165250 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 15:23:33.165675 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.165393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.66:53067)
0924 15:23:33.165613 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.165644 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200}
I20260924 15:23:33.166527 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.166432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:33223)
0924 15:23:33.166472 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.166497 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 15:23:33.166759 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.166615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.65:49277)
0924 15:23:33.166671 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.166728 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 15:23:33.166893 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.166783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.67:38359)
0924 15:23:33.166845 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.166868 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 15:23:33.167097 11886 mvcc.cc:205] Tried to move back new op lower bound from 7332918940318371840 to 7332918940264878080. Current Snapshot: MvccSnapshot[applied={T|T < 7332918940318371840 or (T in {7332918940318371840})}]
I20260924 15:23:33.167335 11855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:33.167248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39321 (local address 127.4.78.66:55915)
0924 15:23:33.167281 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.167306 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:39321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 15:23:33.167608 11316 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:33.167701 11316 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:33.169047 11316 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ce0df8694f9a404ea9292cec4d688550: Sending DeleteTablet for 3 replicas of tablet 01d7614fbda24fb883932e742d5518be
I20260924 15:23:33.169322 11569 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d7614fbda24fb883932e742d5518be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:33 UTC) from {username='oryx', principal='oryx/127.4.78.124@KRBTEST.COM'} at 127.0.0.1:35140
I20260924 15:23:33.169328 11707 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d7614fbda24fb883932e742d5518be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:33 UTC) from {username='oryx', principal='oryx/127.4.78.124@KRBTEST.COM'} at 127.0.0.1:48120
I20260924 15:23:33.169358 11568 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d7614fbda24fb883932e742d5518be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:33 UTC) from {username='oryx', principal='oryx/127.4.78.124@KRBTEST.COM'} at 127.0.0.1:35140
I20260924 15:23:33.169502 11894 tablet_replica.cc:330] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: stopping tablet replica
I20260924 15:23:33.169581 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 11369
I20260924 15:23:33.169642 11433 tablet_service.cc:1514] Processing DeleteTablet for tablet 01d7614fbda24fb883932e742d5518be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:33 UTC) from {username='oryx', principal='oryx/127.4.78.124@KRBTEST.COM'} at 127.0.0.1:34204
I20260924 15:23:33.170482 11895 tablet_replica.cc:330] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: stopping tablet replica
I20260924 15:23:33.170524 11895 raft_consensus.cc:2229] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:33.170552 11895 raft_consensus.cc:2258] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:33.170823 11895 ts_tablet_manager.cc:1905] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:33.172081 11895 ts_tablet_manager.cc:1918] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:33.172128 11895 log.cc:1196] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/01d7614fbda24fb883932e742d5518be
I20260924 15:23:33.172303 11895 ts_tablet_manager.cc:1939] T 01d7614fbda24fb883932e742d5518be P a2d1ee0740124cdbbb87a5f37d1854ea: Deleting consensus metadata
I20260924 15:23:33.172591 11302 catalog_manager.cc:4841] TS a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657): tablet 01d7614fbda24fb883932e742d5518be (table test-table [id=a04203a4245d49bca45fe2106a87889e]) successfully deleted
W20260924 15:23:33.172722 11302 catalog_manager.cc:4808] TS a2d1ee0740124cdbbb87a5f37d1854ea (127.4.78.66:46657): delete failed for tablet 01d7614fbda24fb883932e742d5518be because the tablet was not found. No further retry: Not found: Tablet not found: 01d7614fbda24fb883932e742d5518be
I20260924 15:23:33.174269 11894 raft_consensus.cc:2229] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:33.174362 11894 raft_consensus.cc:2258] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:33.175386 11894 ts_tablet_manager.cc:1905] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 15:23:33.175704 11871 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:33.175213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:37007 (local address 127.0.0.1:34212)
0924 15:23:33.175248 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:33.175257 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:33.175298 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:33.175666 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:37007: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 15:23:33.175868 11301 catalog_manager.cc:4584] TS 70359e1c720b4ac98d0af0a0c1f6e9a5 (127.4.78.65:37007): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 01d7614fbda24fb883932e742d5518be: Network error: Client connection negotiation failed: client connection to 127.4.78.65:37007: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:23:33.175928 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 11506
I20260924 15:23:33.176869 11894 ts_tablet_manager.cc:1918] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:33.176940 11894 log.cc:1196] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/01d7614fbda24fb883932e742d5518be
I20260924 15:23:33.177142 11894 ts_tablet_manager.cc:1939] T 01d7614fbda24fb883932e742d5518be P fee2270f7c7647f89b868163f951feb2: Deleting consensus metadata
I20260924 15:23:33.177592 11302 catalog_manager.cc:4841] TS fee2270f7c7647f89b868163f951feb2 (127.4.78.67:39383): tablet 01d7614fbda24fb883932e742d5518be (table test-table [id=a04203a4245d49bca45fe2106a87889e]) successfully deleted
I20260924 15:23:33.181573 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 11642
I20260924 15:23:33.187191 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 10994
I20260924 15:23:33.191635 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 11184
2026-09-24T15:23:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6570 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:33 dist-test-slave-l766 krb5kdc[11898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:33 dist-test-slave-l766 krb5kdc[11898](info): set up 2 sockets
Sep 24 15:23:33 dist-test-slave-l766 krb5kdc[11898](info): commencing operation
krb5kdc: starting...
W20260924 15:23:34.227449 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.126.keytab.
Entry for principal oryx/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.126.keytab.
I20260924 15:23:34.340337 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:37759
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.126.keytab
--principal=oryx/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:37759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:34.418723 11911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:34.418900 11911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:34.418920 11911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:34.419994 11911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:34.420024 11911 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:23:34.420037 11911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:34.420050 11911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:34.420063 11911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:34.420074 11911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:34.421452 11911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:37759
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.126.keytab
--principal=oryx/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:37759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:34.421622 11911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:34.421785 11911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:23:34.426406 11911 server_base.cc:1034] running on GCE node
W20260924 15:23:34.426357 11917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:34.426384 11919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:34.426339 11916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:34.426883 11911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:34.427139 11911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:34.428277 11911 hybrid_clock.cc:648] HybridClock initialized: now 1790263414428266 us; error 26 us; skew 500 ppm
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:34.430665 11911 init.cc:377] Logged in from keytab as oryx/127.4.78.126@KRBTEST.COM (short username oryx)
I20260924 15:23:34.431123 11911 webserver.cc:460] Webserver started at http://127.4.78.126:44647/ using document root <none> and password file <none>
I20260924 15:23:34.431361 11911 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:34.431428 11911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:34.431573 11911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:34.432554 11911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "0871a092153640dd84db1a9df7c60cc6"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "4b1240df23f088dc00255c92d3f73fef"
server_key_iv: "3ab005a8208d16cdf76cd79ea6c9a13a"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.432904 11911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "0871a092153640dd84db1a9df7c60cc6"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "4b1240df23f088dc00255c92d3f73fef"
server_key_iv: "3ab005a8208d16cdf76cd79ea6c9a13a"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.434221 11911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:34.434886 11926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:34.435052 11911 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:34.435149 11911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "0871a092153640dd84db1a9df7c60cc6"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "4b1240df23f088dc00255c92d3f73fef"
server_key_iv: "3ab005a8208d16cdf76cd79ea6c9a13a"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.435227 11911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:34.465018 11911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:34.466857 11911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:34.467032 11911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:34.472103 11911 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:37759
I20260924 15:23:34.472149 11988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:37759 every 8 connection(s)
I20260924 15:23:34.472503 11911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:34.473132 11989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:34.475273 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 11911
I20260924 15:23:34.475373 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:34.475554 4409 external_mini_cluster.cc:1428] Setting key 61386af509daa2f62a0f76b8f9dd15c5
I20260924 15:23:34.475557 11989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: Bootstrap starting.
I20260924 15:23:34.476241 11989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:34.476533 11989 log.cc:824] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:34.477177 11989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: No bootstrap required, opened a new log
I20260924 15:23:34.478967 11989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871a092153640dd84db1a9df7c60cc6" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37759 } }
I20260924 15:23:34.479127 11989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:34.479230 11989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0871a092153640dd84db1a9df7c60cc6, State: Initialized, Role: FOLLOWER
I20260924 15:23:34.479395 11989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871a092153640dd84db1a9df7c60cc6" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37759 } }
I20260924 15:23:34.479489 11989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:34.479532 11989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:34.479580 11989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:34.480077 11989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871a092153640dd84db1a9df7c60cc6" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37759 } }
I20260924 15:23:34.480195 11989 leader_election.cc:302] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0871a092153640dd84db1a9df7c60cc6; no voters:
I20260924 15:23:34.480376 11989 leader_election.cc:288] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:34.480463 11994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:34.480631 11989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:34.481694 11994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [term 1 LEADER]: Becoming Leader. State: Replica: 0871a092153640dd84db1a9df7c60cc6, State: Running, Role: LEADER
I20260924 15:23:34.481796 11994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871a092153640dd84db1a9df7c60cc6" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37759 } }
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:34.482254 12003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0871a092153640dd84db1a9df7c60cc6. Latest consensus state: current_term: 1 leader_uuid: "0871a092153640dd84db1a9df7c60cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871a092153640dd84db1a9df7c60cc6" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37759 } } }
I20260924 15:23:34.482431 11995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0871a092153640dd84db1a9df7c60cc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0871a092153640dd84db1a9df7c60cc6" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37759 } } }
I20260924 15:23:34.482522 12003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:34.482532 11995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6 [sys.catalog]: This master's current role is: LEADER
W20260924 15:23:34.483108 12008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:23:34.483161 12008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:23:34.483676 12009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:34.483839 12009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:34.485355 12009 catalog_manager.cc:1347] Generated new cluster ID: 1181283132f84bcbb001a2c891443804
I20260924 15:23:34.485411 12009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:34.486370 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.476057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42686 (local address 127.4.78.126:37759)
0924 15:23:34.476245 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:34.476254 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:34.476284 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:23:34.476802 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:34.476810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:34.476871 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:34.477005 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:34.477689 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.478592 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.479464 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.479943 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.480081 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.482827 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:34.482831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:34.482831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:34.482842 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:34.484672 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.485093 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.485096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.485096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.485256 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.485461 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.485461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.485462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.485711 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:34.485907 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:34.486061 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:23:34.486203 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260924 15:23:34.494689 12009 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:34.495185 12009 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:34.500319 12009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: Generated new TSK 0
I20260924 15:23:34.500478 12009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab.
Entry for principal oryx/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab.
I20260924 15:23:34.529084 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37759
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab
--principal=oryx/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:34.631460 12015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:34.631672 12015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:34.631722 12015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:34.632862 12015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:34.632926 12015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:34.632963 12015 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:34.633019 12015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:34.634598 12015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab
--principal=oryx/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:37759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:34.634850 12015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:34.635066 12015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:34.639936 12021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:34.639937 12020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:34.640035 12023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:34.640179 12015 server_base.cc:1034] running on GCE node
I20260924 15:23:34.640547 12015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:34.640810 12015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:34.641958 12015 hybrid_clock.cc:648] HybridClock initialized: now 1790263414641949 us; error 32 us; skew 500 ppm
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:34.644400 12015 init.cc:377] Logged in from keytab as oryx/127.4.78.65@KRBTEST.COM (short username oryx)
I20260924 15:23:34.644822 12015 webserver.cc:460] Webserver started at http://127.4.78.65:38191/ using document root <none> and password file <none>
I20260924 15:23:34.645058 12015 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:34.645102 12015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:34.645254 12015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:34.646360 12015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "667afa2810894b09818663b474f3a90e"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "d7ea4068c8025f56b49603cf475f6b19"
server_key_iv: "f09f76df9374c2f0f2706c1b38eab8d0"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.646713 12015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "667afa2810894b09818663b474f3a90e"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "d7ea4068c8025f56b49603cf475f6b19"
server_key_iv: "f09f76df9374c2f0f2706c1b38eab8d0"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.647997 12015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:34.648705 12030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:34.648880 12015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:34.648978 12015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "667afa2810894b09818663b474f3a90e"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "d7ea4068c8025f56b49603cf475f6b19"
server_key_iv: "f09f76df9374c2f0f2706c1b38eab8d0"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.649071 12015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:34.679271 12015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:34.681644 12015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:34.681798 12015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:34.682410 12015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:34.682475 12015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:34.682513 12015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:34.682534 12015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:34.689512 12015 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:43685
I20260924 15:23:34.689620 12146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:43685 every 8 connection(s)
I20260924 15:23:34.689982 12015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:34.692414 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.682865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:43941 (local address 127.4.78.126:37759)
0924 15:23:34.682940 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:34.682942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:34.683141 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:23:34.683641 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:34.683644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:34.683659 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:34.683745 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:34.684615 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.685482 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.686666 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.687049 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.687145 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.689910 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:34.689912 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:34.689913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:34.689921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:34.691290 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.691651 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.691652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.691652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.691712 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.691910 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.691911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.691911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.692009 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:34.692057 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:34.692266 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:23:34.692334 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:23:34.692557 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.682851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37759 (local address 127.4.78.65:43941)
0924 15:23:34.683007 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:34.683048 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:34.683547 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:34.683837 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:34.683841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:34.684203 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:34.684506 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.684513 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.685589 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.685591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.686594 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.686597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.687187 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.687189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.687345 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.688309 (+ 964us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:34.688318 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:34.689788 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:34.691363 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.691369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.691370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.691585 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.691771 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.691772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.691772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.691858 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.692047 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:34.692050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:34.692165 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 15:23:34.692241 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:23:34.692371 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260924 15:23:34.693097 12147 heartbeater.cc:347] Connected to a master server at 127.4.78.126:37759
I20260924 15:23:34.693181 12147 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:34.693382 12147 heartbeater.cc:511] Master 127.4.78.126:37759 requested a full tablet report, sending...
I20260924 15:23:34.694015 11942 ts_manager.cc:194] Registered new tserver with Master: 667afa2810894b09818663b474f3a90e (127.4.78.65:43685)
I20260924 15:23:34.694684 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 12015
I20260924 15:23:34.694792 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:34.694929 11942 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:43941
I20260924 15:23:34.694979 4409 external_mini_cluster.cc:1428] Setting key fdc06a42e228757c9ebc29e56d754133
I20260924 15:23:34.700325 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.694106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37759 (local address 127.4.78.65:60135)
0924 15:23:34.694244 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:34.694255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:34.694311 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:34.694424 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:34.694424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:34.694614 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:34.694674 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.694675 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.695827 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.695828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.696320 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.696322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.696823 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.696824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.696871 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.697346 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:34.697351 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:34.697627 (+ 276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:34.699374 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.699375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.699375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.699535 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.699760 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.699761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.699761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.699813 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.700089 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:34.700090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:34.700127 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 15:23:34.700179 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:23:34.700259 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260924 15:23:34.700421 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.694172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:60135 (local address 127.4.78.126:37759)
0924 15:23:34.694229 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:34.694230 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:34.694305 (+ 75us) server_negotiation.cc:407] Connection header received
0924 15:23:34.694365 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:34.694365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:34.694373 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:34.694419 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:34.694747 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.695446 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.696432 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.696749 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.696817 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.697724 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:34.697725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:34.697726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:34.697734 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:34.699287 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.699609 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.699611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.699611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.699677 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.699878 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.699879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.699880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.700024 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:34.700097 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:34.700208 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 15:23:34.700292 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
WARNING: no policy specified for oryx/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab.
Entry for principal oryx/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab.
I20260924 15:23:34.720028 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37759
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab
--principal=oryx/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:34.801337 12156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:34.801532 12156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:34.801555 12156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:34.802728 12156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:34.802769 12156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:34.802790 12156 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:34.802809 12156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:34.804316 12156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab
--principal=oryx/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:37759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:34.804555 12156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:34.804776 12156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:34.809429 12162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:34.809439 12164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:34.809430 12161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:34.809751 12156 server_base.cc:1034] running on GCE node
I20260924 15:23:34.809976 12156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:34.810212 12156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:34.811378 12156 hybrid_clock.cc:648] HybridClock initialized: now 1790263414811369 us; error 25 us; skew 500 ppm
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:34.813737 12156 init.cc:377] Logged in from keytab as oryx/127.4.78.66@KRBTEST.COM (short username oryx)
I20260924 15:23:34.814220 12156 webserver.cc:460] Webserver started at http://127.4.78.66:37499/ using document root <none> and password file <none>
I20260924 15:23:34.814448 12156 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:34.814517 12156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:34.814649 12156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:34.815671 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "9c520f5eb7c94c8fbacdac885f19d916"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "ba866a3ee5970234c1691b3c6c8c55e4"
server_key_iv: "578f5201fcffb2548dd7229c93ca94fc"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.816043 12156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "9c520f5eb7c94c8fbacdac885f19d916"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "ba866a3ee5970234c1691b3c6c8c55e4"
server_key_iv: "578f5201fcffb2548dd7229c93ca94fc"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.817332 12156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:23:34.818153 12172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:34.818331 12156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:34.818429 12156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "9c520f5eb7c94c8fbacdac885f19d916"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "ba866a3ee5970234c1691b3c6c8c55e4"
server_key_iv: "578f5201fcffb2548dd7229c93ca94fc"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.818511 12156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:34.835731 12156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:34.837697 12156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:34.837862 12156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:34.838420 12156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:34.838466 12156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:34.838487 12156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:34.838500 12156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:34.845049 12156 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:43165
I20260924 15:23:34.845100 12287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:43165 every 8 connection(s)
I20260924 15:23:34.845388 12156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:34.848500 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.838877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:54583 (local address 127.4.78.126:37759)
0924 15:23:34.838930 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:34.838932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:34.839173 (+ 241us) server_negotiation.cc:407] Connection header received
0924 15:23:34.839714 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:34.839715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:34.839730 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:34.839813 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:34.840631 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.841447 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.843041 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.843396 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.843477 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.846054 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:34.846056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:34.846056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:34.846062 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:34.847478 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.847800 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.847801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.847801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.847849 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.848040 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.848041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.848041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.848138 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:34.848187 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:34.848384 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 15:23:34.848439 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:23:34.848662 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.838839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37759 (local address 127.4.78.66:54583)
0924 15:23:34.839064 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:34.839091 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:34.839574 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:34.839891 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:34.839896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:34.840248 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:34.840531 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.840537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.841545 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.841546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.842947 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.842950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.843485 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.843486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.843629 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.844192 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:34.844202 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:34.845972 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:34.847544 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.847548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.847550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.847738 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.847903 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.847903 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.847904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.847989 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.848193 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:34.848197 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:34.848298 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 15:23:34.848374 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:23:34.848479 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":69,"threads_started":1}
I20260924 15:23:34.849110 12289 heartbeater.cc:347] Connected to a master server at 127.4.78.126:37759
I20260924 15:23:34.849232 12289 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:34.849413 12289 heartbeater.cc:511] Master 127.4.78.126:37759 requested a full tablet report, sending...
I20260924 15:23:34.849831 11943 ts_manager.cc:194] Registered new tserver with Master: 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165)
I20260924 15:23:34.850266 11943 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:54583
I20260924 15:23:34.855073 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 12156
I20260924 15:23:34.855158 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:34.855373 12295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.850168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37759 (local address 127.4.78.66:41195)
0924 15:23:34.850313 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:34.850326 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:34.850386 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:34.850527 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:34.850527 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:34.850735 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:34.850801 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.850802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.851389 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.851389 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.851946 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.851948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.852266 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.852267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.852338 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.852700 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:34.852704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:34.852994 (+ 290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:34.854463 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.854464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.854464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.854606 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.854775 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.854775 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.854776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.854822 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.855074 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:34.855075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:34.855124 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:23:34.855193 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:23:34.855308 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260924 15:23:34.855379 4409 external_mini_cluster.cc:1428] Setting key 90ac4014cfbd281eeb43311646a67fce
I20260924 15:23:34.855417 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.850261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:41195 (local address 127.4.78.126:37759)
0924 15:23:34.850302 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:34.850304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:34.850435 (+ 131us) server_negotiation.cc:407] Connection header received
0924 15:23:34.850456 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:34.850457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:34.850466 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:34.850523 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:34.850873 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.851306 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.852009 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.852180 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.852233 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.853064 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:34.853065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:34.853065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:34.853071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:34.854398 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.854667 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.854667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.854668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.854720 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.854858 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.854859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.854859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.854985 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:34.855065 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:34.855252 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 15:23:34.855329 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for oryx/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab.
Entry for principal oryx/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab.
I20260924 15:23:34.878394 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37759
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab
--principal=oryx/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:34.957283 12298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:34.957460 12298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:34.957482 12298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:34.958608 12298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:34.958644 12298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:34.958664 12298 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:23:34.958688 12298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:34.960062 12298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab
--principal=oryx/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.78.126:37759
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:34.960227 12298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:34.960386 12298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:34.965139 12303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:34.965131 12304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:34.965247 12298 server_base.cc:1034] running on GCE node
W20260924 15:23:34.965131 12307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:34.965590 12298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:34.965807 12298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:34.966958 12298 hybrid_clock.cc:648] HybridClock initialized: now 1790263414966940 us; error 33 us; skew 500 ppm
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:34.969472 12298 init.cc:377] Logged in from keytab as oryx/127.4.78.67@KRBTEST.COM (short username oryx)
I20260924 15:23:34.969960 12298 webserver.cc:460] Webserver started at http://127.4.78.67:35747/ using document root <none> and password file <none>
I20260924 15:23:34.970185 12298 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:34.970230 12298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:34.970367 12298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:34.971319 12298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "b32072fd474c445d82eed389283e2a2e"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "1f85aa125b1e87d3447184bbdaba4f4c"
server_key_iv: "ee0ae84ed3b116ead5f823a130ecdf5a"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.971642 12298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "b32072fd474c445d82eed389283e2a2e"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "1f85aa125b1e87d3447184bbdaba4f4c"
server_key_iv: "ee0ae84ed3b116ead5f823a130ecdf5a"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.972887 12298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:23:34.973558 12314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:34.973726 12298 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:34.973835 12298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "b32072fd474c445d82eed389283e2a2e"
format_stamp: "Formatted at 2026-09-24 15:23:34 on dist-test-slave-l766"
server_key: "1f85aa125b1e87d3447184bbdaba4f4c"
server_key_iv: "ee0ae84ed3b116ead5f823a130ecdf5a"
server_key_version: "encryptionkey@0"
I20260924 15:23:34.973917 12298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:34.986842 12298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:34.988509 12298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:34.988648 12298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:34.989176 12298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:34.989240 12298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:34.989276 12298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:34.989303 12298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:23:34 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:34.996658 12298 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:43149
I20260924 15:23:34.996779 12429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:43149 every 8 connection(s)
I20260924 15:23:34.997030 12298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:34.998651 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.989662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:36923 (local address 127.4.78.126:37759)
0924 15:23:34.989732 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:34.989735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:34.989889 (+ 154us) server_negotiation.cc:407] Connection header received
0924 15:23:34.990341 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:34.990343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:34.990357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:34.990434 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:34.991010 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.991512 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.992631 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.992920 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.993006 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.995811 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:34.995813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:34.995813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:34.995822 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:34.997582 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.997989 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.997989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.997990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.998041 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:34.998218 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:34.998218 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:34.998218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:34.998309 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:34.998353 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:34.998524 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 15:23:34.998589 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:23:34.998796 12323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:34.989601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37759 (local address 127.4.78.67:36923)
0924 15:23:34.989755 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:34.989793 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:34.990250 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:34.990494 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:34.990497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:34.990718 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:34.990908 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.990912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.991606 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.991607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.992539 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:34.992542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:34.993018 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:34.993019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:34.993229 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:34.993731 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:34.993747 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:34.995693 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:34.997648 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.997651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.997652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.997925 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.998090 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:34.998091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:34.998091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:34.998173 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:34.998343 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:34.998346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:34.998442 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 15:23:34.998506 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:23:34.998616 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 15:23:34.999229 12430 heartbeater.cc:347] Connected to a master server at 127.4.78.126:37759
I20260924 15:23:34.999325 12430 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:34.999490 12430 heartbeater.cc:511] Master 127.4.78.126:37759 requested a full tablet report, sending...
I20260924 15:23:34.999866 11942 ts_manager.cc:194] Registered new tserver with Master: b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149)
I20260924 15:23:35.000363 11942 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:36923
I20260924 15:23:35.002671 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 12298
I20260924 15:23:35.002779 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:35.002959 4409 external_mini_cluster.cc:1428] Setting key 35af80387134adf96e5bae91f0906566
I20260924 15:23:35.003779 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:23:35.005733 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.000245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37759 (local address 127.4.78.67:53161)
0924 15:23:35.000366 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.000398 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.000463 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.000588 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.000589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.000800 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:35.000864 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.000865 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.001458 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.001458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.002006 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.002008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.002311 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.002311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.002360 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.002773 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:35.002778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:35.003110 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:35.004897 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:35.004897 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:35.004898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:35.005051 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:35.005256 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:35.005257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:35.005257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:35.005298 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:35.005518 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:35.005519 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:35.005553 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.005603 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.005673 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260924 15:23:35.005853 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.000285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:53161 (local address 127.4.78.126:37759)
0924 15:23:35.000323 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.000325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.000484 (+ 159us) server_negotiation.cc:407] Connection header received
0924 15:23:35.000510 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.000511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.000520 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.000584 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:35.000932 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.001372 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.002072 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.002249 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.002299 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.003195 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:35.003197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:35.003197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:35.003204 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:35.004813 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:35.005122 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:35.005123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:35.005123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:35.005182 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:35.005333 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:35.005333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:35.005334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:35.005460 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:35.005520 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.005630 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.005706 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263415, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:23:35.025146 11992 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:35.019747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42694 (local address 127.4.78.126:37759)
0924 15:23:35.019813 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.019815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.019825 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:35.019893 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.019894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.019910 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.019971 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:35.020533 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.021012 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.021797 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.022020 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.022123 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.025033 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42694: BlockingRecv error: recv got EOF from 127.0.0.1:42694 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263415, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:35.031955 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.025842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.4.78.126:37759)
0924 15:23:35.025888 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.025890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.025947 (+ 57us) server_negotiation.cc:407] Connection header received
0924 15:23:35.025992 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.025992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.026000 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.026048 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:35.026479 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.026917 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.027542 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.027711 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.027758 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.029708 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:35.029710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:35.029710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:35.029716 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:35.031137 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:35.031423 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:35.031424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:35.031424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:35.031472 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:35.031615 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:35.031615 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:35.031615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:35.031698 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:35.031768 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.031845 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.031897 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:23:35.033542 11942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:35.035154 11942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:35.041931 12452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.037459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43685 (local address 127.0.0.1:55910)
0924 15:23:35.037585 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.037633 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.037703 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.038344 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.038345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.038349 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:35.038410 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.038411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.039773 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.039774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.040684 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.040686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.041610 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.041611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.041677 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.041682 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.041714 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.041781 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":6,"thread_start_us":59,"threads_started":1}
I20260924 15:23:35.041947 12454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.037674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43165 (local address 127.0.0.1:34216)
0924 15:23:35.037965 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.037975 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.038037 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.038258 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.038262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.038263 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:35.038330 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.038331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.039496 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.039497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.040785 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.040788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.041523 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.041529 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.041640 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.041653 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.041754 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.041825 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":64,"thread_start_us":70,"threads_started":1}
I20260924 15:23:35.042209 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.037779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.4.78.66:43165)
0924 15:23:35.037954 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.037959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.038023 (+ 64us) server_negotiation.cc:407] Connection header received
0924 15:23:35.038106 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.038109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.038125 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.038245 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:35.038495 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.039406 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.040886 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.041428 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.041516 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.041600 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.042050 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.042103 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260924 15:23:35.042333 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.037969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54680 (local address 127.4.78.67:43149)
0924 15:23:35.038148 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.038154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.038166 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:23:35.038251 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.038253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.038268 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.038339 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:35.038497 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.039639 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.040953 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.041649 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.041725 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.041801 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.041887 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.041926 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260924 15:23:35.042335 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.037619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.4.78.65:43685)
0924 15:23:35.037875 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.037881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.037893 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:23:35.038151 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.038153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.038169 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.038485 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:35.038546 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.039672 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.040774 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.041504 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.042065 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.042158 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.042182 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.042231 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":78,"threads_started":1}
I20260924 15:23:35.042506 12456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.037958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:43149 (local address 127.0.0.1:54680)
0924 15:23:35.038125 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.038134 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.038171 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.038344 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.038345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.038347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:35.038410 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.038411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.039747 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.039747 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.040875 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.040877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.041729 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.041730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.041795 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.041798 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.041858 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.041894 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":41,"thread_start_us":66,"threads_started":1}
I20260924 15:23:35.042897 12222 tablet_service.cc:1467] Processing CreateTablet for tablet a74dea6a736a4250ae1fb8c376754359 (DEFAULT_TABLE table=test-table [id=bc3a6e85edde46989a8e6927ecc882d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.043123 12364 tablet_service.cc:1467] Processing CreateTablet for tablet a74dea6a736a4250ae1fb8c376754359 (DEFAULT_TABLE table=test-table [id=bc3a6e85edde46989a8e6927ecc882d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.043274 12222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74dea6a736a4250ae1fb8c376754359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.043452 12364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74dea6a736a4250ae1fb8c376754359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.045960 12458 tablet_bootstrap.cc:492] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: Bootstrap starting.
I20260924 15:23:35.046211 12081 tablet_service.cc:1467] Processing CreateTablet for tablet a74dea6a736a4250ae1fb8c376754359 (DEFAULT_TABLE table=test-table [id=bc3a6e85edde46989a8e6927ecc882d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.046576 12081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a74dea6a736a4250ae1fb8c376754359. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.046828 12458 tablet_bootstrap.cc:654] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.047163 12458 log.cc:824] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:35.047926 12458 tablet_bootstrap.cc:492] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: No bootstrap required, opened a new log
I20260924 15:23:35.048002 12458 ts_tablet_manager.cc:1397] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:35.050504 12460 tablet_bootstrap.cc:492] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: Bootstrap starting.
I20260924 15:23:35.050611 12461 tablet_bootstrap.cc:492] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Bootstrap starting.
I20260924 15:23:35.050364 12458 raft_consensus.cc:348] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.050742 12458 raft_consensus.cc:374] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.050791 12458 raft_consensus.cc:729] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32072fd474c445d82eed389283e2a2e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.050918 12458 consensus_queue.cc:260] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.051229 12458 ts_tablet_manager.cc:1428] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:23:35.051267 12461 tablet_bootstrap.cc:654] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.051357 12430 heartbeater.cc:503] Master 127.4.78.126:37759 was elected leader, sending a full tablet report...
I20260924 15:23:35.051368 12460 tablet_bootstrap.cc:654] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.051581 12461 log.cc:824] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:35.051690 12460 log.cc:824] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:35.052500 12460 tablet_bootstrap.cc:492] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: No bootstrap required, opened a new log
I20260924 15:23:35.052531 12461 tablet_bootstrap.cc:492] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: No bootstrap required, opened a new log
I20260924 15:23:35.052590 12460 ts_tablet_manager.cc:1397] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:23:35.052604 12461 ts_tablet_manager.cc:1397] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:35.054597 12461 raft_consensus.cc:348] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.054597 12460 raft_consensus.cc:348] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.054754 12460 raft_consensus.cc:374] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.054754 12461 raft_consensus.cc:374] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.054790 12461 raft_consensus.cc:729] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c520f5eb7c94c8fbacdac885f19d916, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.054790 12460 raft_consensus.cc:729] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667afa2810894b09818663b474f3a90e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.054908 12460 consensus_queue.cc:260] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.054908 12461 consensus_queue.cc:260] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.055153 12460 ts_tablet_manager.cc:1428] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:23:35.055153 12461 ts_tablet_manager.cc:1428] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:23:35.055207 12289 heartbeater.cc:503] Master 127.4.78.126:37759 was elected leader, sending a full tablet report...
I20260924 15:23:35.055279 12147 heartbeater.cc:503] Master 127.4.78.126:37759 was elected leader, sending a full tablet report...
W20260924 15:23:35.095986 12290 tablet.cc:2369] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:35.147639 12466 raft_consensus.cc:482] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:35.147768 12466 raft_consensus.cc:504] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.148066 12466 leader_election.cc:288] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165), b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149)
I20260924 15:23:35.151038 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.148367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:42175 (local address 127.4.78.66:43165)
0924 15:23:35.148434 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.148437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.148447 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:35.148475 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.148476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.148488 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.148560 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:35.148761 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.149314 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.150331 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.150701 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.150784 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.150795 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.150936 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.150971 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:23:35.151059 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.148261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43165 (local address 127.4.78.65:42175)
0924 15:23:35.148321 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.148334 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.148380 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.148565 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.148567 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.148569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:35.148643 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.148645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.149401 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.149402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.150244 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.150247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.150783 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.150784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.150861 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.150864 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.150931 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.150971 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":5}
I20260924 15:23:35.151260 12467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.148391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:43149 (local address 127.4.78.65:59143)
0924 15:23:35.148627 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.148636 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.148715 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.148866 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.148867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.148869 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:35.148940 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.148944 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.149606 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.149608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.150403 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.150406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.150937 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.150938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.151013 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.151016 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.151090 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.151136 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":143,"threads_started":1}
I20260924 15:23:35.151353 12457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.148488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:59143 (local address 127.4.78.67:43149)
0924 15:23:35.148555 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.148557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.148744 (+ 187us) server_negotiation.cc:407] Connection header received
0924 15:23:35.148774 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.148775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.148787 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.148863 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:35.149023 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.149526 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.150503 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.150854 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.150936 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.150944 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.151233 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.151275 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:23:35.151383 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74dea6a736a4250ae1fb8c376754359" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916" is_pre_election: true
I20260924 15:23:35.151539 12242 raft_consensus.cc:2454] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 0.
I20260924 15:23:35.151801 12034 leader_election.cc:302] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, 9c520f5eb7c94c8fbacdac885f19d916; no voters:
I20260924 15:23:35.151962 12466 raft_consensus.cc:2781] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:35.151965 12384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74dea6a736a4250ae1fb8c376754359" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32072fd474c445d82eed389283e2a2e" is_pre_election: true
I20260924 15:23:35.152037 12466 raft_consensus.cc:482] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:35.152070 12466 raft_consensus.cc:3037] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.152112 12384 raft_consensus.cc:2454] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 0.
I20260924 15:23:35.152868 12466 raft_consensus.cc:504] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.153018 12466 leader_election.cc:288] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 election: Requested vote from peers 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165), b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149)
I20260924 15:23:35.153155 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74dea6a736a4250ae1fb8c376754359" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916"
I20260924 15:23:35.153178 12384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a74dea6a736a4250ae1fb8c376754359" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32072fd474c445d82eed389283e2a2e"
I20260924 15:23:35.153234 12242 raft_consensus.cc:3037] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.153275 12384 raft_consensus.cc:3037] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.153980 12242 raft_consensus.cc:2454] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 1.
I20260924 15:23:35.154011 12384 raft_consensus.cc:2454] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 1.
I20260924 15:23:35.154163 12034 leader_election.cc:302] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, b32072fd474c445d82eed389283e2a2e; no voters:
I20260924 15:23:35.154282 12466 raft_consensus.cc:2781] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:35.154434 12466 raft_consensus.cc:686] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 1 LEADER]: Becoming Leader. State: Replica: 667afa2810894b09818663b474f3a90e, State: Running, Role: LEADER
I20260924 15:23:35.154537 12466 consensus_queue.cc:237] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.155326 11942 catalog_manager.cc:5495] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e reported cstate change: term changed from 0 to 1, leader changed from <none> to 667afa2810894b09818663b474f3a90e (127.4.78.65). New cstate: current_term: 1 leader_uuid: "667afa2810894b09818663b474f3a90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } health_report { overall_health: UNKNOWN } } }
I20260924 15:23:35.185287 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.182861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55922 (local address 127.4.78.65:43685)
0924 15:23:35.182905 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.182907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.182915 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:35.182984 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.182986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.183011 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.183089 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:35.183265 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.183708 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.184479 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.184670 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.184761 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.184967 (+ 206us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:35.185067 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:35.185161 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.185187 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.185206 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:23:35.187371 12242 raft_consensus.cc:1261] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Refusing update from remote peer 667afa2810894b09818663b474f3a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:35.187412 12384 raft_consensus.cc:1261] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Refusing update from remote peer 667afa2810894b09818663b474f3a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:35.187696 12468 consensus_queue.cc:1035] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:35.187824 12466 consensus_queue.cc:1035] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:35.190634 12472 mvcc.cc:205] Tried to move back new op lower bound from 7332918948604973056 to 7332918948473511936. Current Snapshot: MvccSnapshot[applied={T|T < 7332918948604973056 or (T in {7332918948604973056})}]
I20260924 15:23:35.191035 12473 mvcc.cc:205] Tried to move back new op lower bound from 7332918948604973056 to 7332918948473511936. Current Snapshot: MvccSnapshot[applied={T|T < 7332918948604973056}]
I20260924 15:23:35.192425 11942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:35.192531 11942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:35.193323 11942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: Sending DeleteTablet for 3 replicas of tablet a74dea6a736a4250ae1fb8c376754359
I20260924 15:23:35.193667 12080 tablet_service.cc:1514] Processing DeleteTablet for tablet a74dea6a736a4250ae1fb8c376754359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:55910
I20260924 15:23:35.193727 12081 tablet_service.cc:1514] Processing DeleteTablet for tablet a74dea6a736a4250ae1fb8c376754359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:55910
I20260924 15:23:35.193786 11942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 15:23:35.193852 12364 tablet_service.cc:1514] Processing DeleteTablet for tablet a74dea6a736a4250ae1fb8c376754359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:54680
W20260924 15:23:35.193955 11942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:35.194110 12222 tablet_service.cc:1514] Processing DeleteTablet for tablet a74dea6a736a4250ae1fb8c376754359 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:34216
I20260924 15:23:35.194311 12481 tablet_replica.cc:330] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: stopping tablet replica
I20260924 15:23:35.194407 12481 raft_consensus.cc:2229] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:35.194504 12481 raft_consensus.cc:2258] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.194867 12483 tablet_replica.cc:330] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: stopping tablet replica
I20260924 15:23:35.194967 12483 raft_consensus.cc:2229] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:35.195075 12483 raft_consensus.cc:2258] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.195412 12080 tablet_service.cc:1467] Processing CreateTablet for tablet 3e38c9c6cd654c88acce08ec188083b1 (DEFAULT_TABLE table=test-table [id=4c7061132c074a6faed35bdb86334c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.195538 12080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e38c9c6cd654c88acce08ec188083b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.195618 12481 ts_tablet_manager.cc:1905] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.195811 12222 tablet_service.cc:1467] Processing CreateTablet for tablet 3e38c9c6cd654c88acce08ec188083b1 (DEFAULT_TABLE table=test-table [id=4c7061132c074a6faed35bdb86334c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.195909 12222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e38c9c6cd654c88acce08ec188083b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.196965 12460 tablet_bootstrap.cc:492] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: Bootstrap starting.
I20260924 15:23:35.197304 12461 tablet_bootstrap.cc:492] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: Bootstrap starting.
I20260924 15:23:35.197300 12481 ts_tablet_manager.cc:1918] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:35.197366 12481 log.cc:1196] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/a74dea6a736a4250ae1fb8c376754359
I20260924 15:23:35.197393 12460 tablet_bootstrap.cc:654] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.197633 12481 ts_tablet_manager.cc:1939] T a74dea6a736a4250ae1fb8c376754359 P 667afa2810894b09818663b474f3a90e: Deleting consensus metadata
I20260924 15:23:35.197723 12461 tablet_bootstrap.cc:654] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.198122 12483 ts_tablet_manager.cc:1905] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.197264 12364 tablet_service.cc:1467] Processing CreateTablet for tablet 3e38c9c6cd654c88acce08ec188083b1 (DEFAULT_TABLE table=test-table [id=4c7061132c074a6faed35bdb86334c1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.198426 12364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e38c9c6cd654c88acce08ec188083b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.199633 12483 ts_tablet_manager.cc:1918] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:35.199703 12483 log.cc:1196] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/a74dea6a736a4250ae1fb8c376754359
I20260924 15:23:35.199957 12458 tablet_bootstrap.cc:492] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: Bootstrap starting.
I20260924 15:23:35.199985 12483 ts_tablet_manager.cc:1939] T a74dea6a736a4250ae1fb8c376754359 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting consensus metadata
I20260924 15:23:35.200399 12461 tablet_bootstrap.cc:492] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: No bootstrap required, opened a new log
I20260924 15:23:35.200402 12458 tablet_bootstrap.cc:654] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.200443 12461 ts_tablet_manager.cc:1397] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 15:23:35.200436 11930 catalog_manager.cc:4841] TS 667afa2810894b09818663b474f3a90e (127.4.78.65:43685): tablet a74dea6a736a4250ae1fb8c376754359 (table test-table [id=bc3a6e85edde46989a8e6927ecc882d2]) successfully deleted
I20260924 15:23:35.200588 12461 raft_consensus.cc:348] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.200654 12461 raft_consensus.cc:374] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.200678 12461 raft_consensus.cc:729] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c520f5eb7c94c8fbacdac885f19d916, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.200719 12461 consensus_queue.cc:260] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.200737 11930 catalog_manager.cc:4841] TS 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165): tablet a74dea6a736a4250ae1fb8c376754359 (table test-table [id=bc3a6e85edde46989a8e6927ecc882d2]) successfully deleted
I20260924 15:23:35.200812 12461 ts_tablet_manager.cc:1428] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:35.194293 12482 tablet_replica.cc:330] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: stopping tablet replica
W20260924 15:23:35.201013 11930 catalog_manager.cc:4808] TS 667afa2810894b09818663b474f3a90e (127.4.78.65:43685): delete failed for tablet a74dea6a736a4250ae1fb8c376754359 because the tablet was not found. No further retry: Not found: Tablet not found: a74dea6a736a4250ae1fb8c376754359
I20260924 15:23:35.201057 12482 raft_consensus.cc:2229] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:35.201319 12482 raft_consensus.cc:2258] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.201771 12482 ts_tablet_manager.cc:1905] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.201785 12460 tablet_bootstrap.cc:492] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: No bootstrap required, opened a new log
I20260924 15:23:35.201851 12460 ts_tablet_manager.cc:1397] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 15:23:35.201980 12460 raft_consensus.cc:348] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.202036 12460 raft_consensus.cc:374] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.202055 12460 raft_consensus.cc:729] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667afa2810894b09818663b474f3a90e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.202107 12460 consensus_queue.cc:260] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.202188 12460 ts_tablet_manager.cc:1428] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:35.203385 12482 ts_tablet_manager.cc:1918] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:35.203444 12482 log.cc:1196] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/a74dea6a736a4250ae1fb8c376754359
I20260924 15:23:35.203613 12482 ts_tablet_manager.cc:1939] T a74dea6a736a4250ae1fb8c376754359 P b32072fd474c445d82eed389283e2a2e: Deleting consensus metadata
I20260924 15:23:35.203930 11930 catalog_manager.cc:4841] TS b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149): tablet a74dea6a736a4250ae1fb8c376754359 (table test-table [id=bc3a6e85edde46989a8e6927ecc882d2]) successfully deleted
I20260924 15:23:35.206188 12458 tablet_bootstrap.cc:492] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: No bootstrap required, opened a new log
I20260924 15:23:35.206251 12458 ts_tablet_manager.cc:1397] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260924 15:23:35.206382 12458 raft_consensus.cc:348] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.206455 12458 raft_consensus.cc:374] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.206476 12458 raft_consensus.cc:729] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32072fd474c445d82eed389283e2a2e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.206516 12458 consensus_queue.cc:260] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.206597 12458 ts_tablet_manager.cc:1428] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260924 15:23:35.247700 12431 tablet.cc:2369] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:35.364236 12478 raft_consensus.cc:482] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:35.364329 12478 raft_consensus.cc:504] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.364470 12478 leader_election.cc:288] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165), b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149)
I20260924 15:23:35.364720 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e38c9c6cd654c88acce08ec188083b1" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916" is_pre_election: true
I20260924 15:23:35.364825 12242 raft_consensus.cc:2454] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 0.
I20260924 15:23:35.364776 12384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e38c9c6cd654c88acce08ec188083b1" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32072fd474c445d82eed389283e2a2e" is_pre_election: true
I20260924 15:23:35.364888 12384 raft_consensus.cc:2454] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 0.
I20260924 15:23:35.365005 12034 leader_election.cc:302] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, 9c520f5eb7c94c8fbacdac885f19d916; no voters:
I20260924 15:23:35.365120 12478 raft_consensus.cc:2781] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:35.365162 12478 raft_consensus.cc:482] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:35.365199 12478 raft_consensus.cc:3037] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.365963 12478 raft_consensus.cc:504] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.366087 12478 leader_election.cc:288] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 election: Requested vote from peers 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165), b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149)
I20260924 15:23:35.366230 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e38c9c6cd654c88acce08ec188083b1" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916"
I20260924 15:23:35.366273 12384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e38c9c6cd654c88acce08ec188083b1" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32072fd474c445d82eed389283e2a2e"
I20260924 15:23:35.366325 12384 raft_consensus.cc:3037] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.366331 12242 raft_consensus.cc:3037] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.366933 12242 raft_consensus.cc:2454] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 1.
I20260924 15:23:35.366933 12384 raft_consensus.cc:2454] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 1.
I20260924 15:23:35.367128 12034 leader_election.cc:302] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, b32072fd474c445d82eed389283e2a2e; no voters:
I20260924 15:23:35.367219 12478 raft_consensus.cc:2781] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:35.367274 12478 raft_consensus.cc:686] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 1 LEADER]: Becoming Leader. State: Replica: 667afa2810894b09818663b474f3a90e, State: Running, Role: LEADER
I20260924 15:23:35.367329 12478 consensus_queue.cc:237] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.367926 11942 catalog_manager.cc:5495] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e reported cstate change: term changed from 0 to 1, leader changed from <none> to 667afa2810894b09818663b474f3a90e (127.4.78.65). New cstate: current_term: 1 leader_uuid: "667afa2810894b09818663b474f3a90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } health_report { overall_health: UNKNOWN } } }
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263414, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.126@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:35.417213 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.410499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37759 (local address 127.0.0.1:42708)
0924 15:23:35.410701 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.410714 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.410784 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.410911 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.410911 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.411232 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:35.411291 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.411293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.411898 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.411899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.412439 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.412440 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.412737 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.412738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.412778 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.413362 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:35.413367 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:35.414730 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:35.416042 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:35.416049 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:35.416051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:35.416278 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:35.416461 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:35.416462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:35.416462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:35.416515 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:35.416924 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:35.416929 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:35.416987 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.417043 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.417128 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":99,"threads_started":1}
I20260924 15:23:35.417309 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.410555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42708 (local address 127.4.78.126:37759)
0924 15:23:35.410619 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.410621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.410780 (+ 159us) server_negotiation.cc:407] Connection header received
0924 15:23:35.410827 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.410827 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.410837 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.410887 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:35.411353 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.411834 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.412505 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.412676 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.412724 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.414828 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:35.414829 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:35.414830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:35.414836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:35.415974 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:35.416348 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:35.416349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:35.416349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:35.416401 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:35.416726 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:35.416728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:35.416728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:35.416851 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:35.416913 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.417094 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.417177 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":36}
I20260924 15:23:35.419118 11942 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:42708:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:23:35.419323 11942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:35.420723 12222 tablet_service.cc:1467] Processing CreateTablet for tablet 582d8df8cf8c430c91d19357b391df17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=502900aa591b4b6fa412075cf9fb69f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:35.420727 12080 tablet_service.cc:1467] Processing CreateTablet for tablet 582d8df8cf8c430c91d19357b391df17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=502900aa591b4b6fa412075cf9fb69f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:35.420890 12222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582d8df8cf8c430c91d19357b391df17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.420890 12080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582d8df8cf8c430c91d19357b391df17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.422521 12460 tablet_bootstrap.cc:492] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: Bootstrap starting.
I20260924 15:23:35.422521 12461 tablet_bootstrap.cc:492] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: Bootstrap starting.
I20260924 15:23:35.422606 12364 tablet_service.cc:1467] Processing CreateTablet for tablet 582d8df8cf8c430c91d19357b391df17 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=502900aa591b4b6fa412075cf9fb69f2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:23:35.422736 12364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582d8df8cf8c430c91d19357b391df17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.422883 12460 tablet_bootstrap.cc:654] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.422917 12461 tablet_bootstrap.cc:654] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.423480 12460 tablet_bootstrap.cc:492] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: No bootstrap required, opened a new log
I20260924 15:23:35.423543 12460 ts_tablet_manager.cc:1397] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:35.423666 12460 raft_consensus.cc:348] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.423786 12460 raft_consensus.cc:374] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.423812 12460 raft_consensus.cc:729] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667afa2810894b09818663b474f3a90e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.423861 12460 consensus_queue.cc:260] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.423957 12460 ts_tablet_manager.cc:1428] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:35.423972 12478 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.424053 12478 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:35.424134 12458 tablet_bootstrap.cc:492] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: Bootstrap starting.
I20260924 15:23:35.424571 12458 tablet_bootstrap.cc:654] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.424618 12461 tablet_bootstrap.cc:492] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: No bootstrap required, opened a new log
I20260924 15:23:35.424659 12461 ts_tablet_manager.cc:1397] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:23:35.424835 12461 raft_consensus.cc:348] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.424921 12461 raft_consensus.cc:374] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.424947 12461 raft_consensus.cc:729] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c520f5eb7c94c8fbacdac885f19d916, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.424998 12461 consensus_queue.cc:260] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.425098 12461 ts_tablet_manager.cc:1428] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:35.425168 12465 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.425266 12465 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:35.425329 12458 tablet_bootstrap.cc:492] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: No bootstrap required, opened a new log
I20260924 15:23:35.425376 12458 ts_tablet_manager.cc:1397] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:35.425523 12458 raft_consensus.cc:348] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.425594 12458 raft_consensus.cc:374] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.425618 12458 raft_consensus.cc:729] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32072fd474c445d82eed389283e2a2e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.425664 12458 consensus_queue.cc:260] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.425755 12458 ts_tablet_manager.cc:1428] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:35.425843 12462 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.425961 12462 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 15:23:35.441008 12148 tablet.cc:2369] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:35.639513 12462 raft_consensus.cc:482] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:35.639602 12462 raft_consensus.cc:504] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.639897 12462 leader_election.cc:288] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 667afa2810894b09818663b474f3a90e (127.4.78.65:43685), 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165)
I20260924 15:23:35.642967 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.640172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:56055 (local address 127.4.78.65:43685)
0924 15:23:35.640258 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.640261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.640431 (+ 170us) server_negotiation.cc:407] Connection header received
0924 15:23:35.640479 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.640479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.640487 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.640551 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:35.640703 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.641180 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.642265 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.642627 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.642706 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.642718 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.642859 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.642898 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 15:23:35.642972 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.640392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:55329 (local address 127.4.78.66:43165)
0924 15:23:35.640461 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.640463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.640473 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:35.640538 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.640539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.640547 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.640618 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:35.640772 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.641240 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.642259 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.642611 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.642706 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.642718 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.642866 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.642901 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:23:35.642997 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.640300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43165 (local address 127.4.78.67:55329)
0924 15:23:35.640381 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.640397 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.640469 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.640621 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.640622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.640624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:35.640698 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.640699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.641318 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.641319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.642158 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.642160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.642713 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.642714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.642781 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.642785 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.642858 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.642900 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"thread_start_us":39,"threads_started":1}
I20260924 15:23:35.643014 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.640096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43685 (local address 127.4.78.67:56055)
0924 15:23:35.640250 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.640262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.640381 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.640557 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.640558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.640561 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:35.640621 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.640623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.641276 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.641277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.642167 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.642170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.642706 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.642707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.642771 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.642779 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.642841 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.642885 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":42,"thread_start_us":77,"threads_started":1}
I20260924 15:23:35.643249 12101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582d8df8cf8c430c91d19357b391df17" candidate_uuid: "b32072fd474c445d82eed389283e2a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667afa2810894b09818663b474f3a90e" is_pre_election: true
I20260924 15:23:35.643385 12101 raft_consensus.cc:2454] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32072fd474c445d82eed389283e2a2e in term 0.
I20260924 15:23:35.643388 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582d8df8cf8c430c91d19357b391df17" candidate_uuid: "b32072fd474c445d82eed389283e2a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916" is_pre_election: true
I20260924 15:23:35.643486 12242 raft_consensus.cc:2454] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32072fd474c445d82eed389283e2a2e in term 0.
I20260924 15:23:35.643581 12318 leader_election.cc:302] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, b32072fd474c445d82eed389283e2a2e; no voters:
I20260924 15:23:35.643726 12462 raft_consensus.cc:2781] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:35.643796 12462 raft_consensus.cc:482] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:35.643828 12462 raft_consensus.cc:3037] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.644531 12462 raft_consensus.cc:504] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.644670 12462 leader_election.cc:288] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [CANDIDATE]: Term 1 election: Requested vote from peers 667afa2810894b09818663b474f3a90e (127.4.78.65:43685), 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165)
I20260924 15:23:35.644832 12101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582d8df8cf8c430c91d19357b391df17" candidate_uuid: "b32072fd474c445d82eed389283e2a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667afa2810894b09818663b474f3a90e"
I20260924 15:23:35.644914 12101 raft_consensus.cc:3037] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.644856 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "582d8df8cf8c430c91d19357b391df17" candidate_uuid: "b32072fd474c445d82eed389283e2a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916"
I20260924 15:23:35.644944 12242 raft_consensus.cc:3037] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.645757 12242 raft_consensus.cc:2454] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32072fd474c445d82eed389283e2a2e in term 1.
I20260924 15:23:35.645782 12101 raft_consensus.cc:2454] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32072fd474c445d82eed389283e2a2e in term 1.
I20260924 15:23:35.645936 12318 leader_election.cc:302] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c520f5eb7c94c8fbacdac885f19d916, b32072fd474c445d82eed389283e2a2e; no voters:
I20260924 15:23:35.646031 12462 raft_consensus.cc:2781] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:35.646155 12462 raft_consensus.cc:686] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [term 1 LEADER]: Becoming Leader. State: Replica: b32072fd474c445d82eed389283e2a2e, State: Running, Role: LEADER
I20260924 15:23:35.646236 12462 consensus_queue.cc:237] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.646745 12499 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: TxnStatusTablet state changed. Reason: New leader b32072fd474c445d82eed389283e2a2e. Latest consensus state: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.646893 12499 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:35.646984 11942 catalog_manager.cc:5495] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e reported cstate change: term changed from 0 to 1, leader changed from <none> to b32072fd474c445d82eed389283e2a2e (127.4.78.67). New cstate: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } health_report { overall_health: HEALTHY } } }
I20260924 15:23:35.647081 12501 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:23:35.647213 12501 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:23:35.706954 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.704062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.4.78.67:43149)
0924 15:23:35.704155 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.704157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.704166 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:35.704213 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.704214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.704231 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.704316 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:35.704494 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.704976 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.706120 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.706350 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.706403 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.706635 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:35.706727 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:35.706781 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.706870 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.706888 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":57,"threads_started":1}
I20260924 15:23:35.707010 12493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.703891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:43149 (local address 127.0.0.1:54694)
0924 15:23:35.703976 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.703989 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.704035 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.704357 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.704358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.704359 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:35.704419 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.704422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.705061 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.705062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.706012 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.706015 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.706432 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.706433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.706506 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.706525 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:35.706795 (+ 270us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:35.706806 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.706868 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.706922 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 15:23:35.708259 12242 raft_consensus.cc:1261] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Refusing update from remote peer b32072fd474c445d82eed389283e2a2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:35.708529 12101 raft_consensus.cc:1261] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Refusing update from remote peer b32072fd474c445d82eed389283e2a2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:35.708621 12499 consensus_queue.cc:1035] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:35.708799 12499 consensus_queue.cc:1035] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:35.709259 12465 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: TxnStatusTablet state changed. Reason: New leader b32072fd474c445d82eed389283e2a2e. Latest consensus state: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.709349 12465 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:35.710067 12462 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.710160 12462 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:35.710322 12478 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: TxnStatusTablet state changed. Reason: New leader b32072fd474c445d82eed389283e2a2e. Latest consensus state: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.710408 12478 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:35.710829 12465 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.710914 12465 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P 9c520f5eb7c94c8fbacdac885f19d916: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:35.711727 12499 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.711807 12499 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P b32072fd474c445d82eed389283e2a2e: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:23:35.712066 12478 tablet_replica.cc:439] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } } }
I20260924 15:23:35.712152 12478 tablet_replica.cc:442] T 582d8df8cf8c430c91d19357b391df17 P 667afa2810894b09818663b474f3a90e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:23:35.716274 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.713912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:50599 (local address 127.4.78.67:43149)
0924 15:23:35.713949 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.713951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.714088 (+ 137us) server_negotiation.cc:407] Connection header received
0924 15:23:35.714155 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.714156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.714166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.714221 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:35.714326 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.714780 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.715536 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.715743 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.715838 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.716014 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:35.716051 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:35.716127 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.716189 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.716210 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:23:35.716317 12516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.713859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:43149 (local address 127.4.78.65:50599)
0924 15:23:35.714020 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.714031 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.714103 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.714224 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.714225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.714227 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:35.714271 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.714273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.714862 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.714863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.715451 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.715453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.715841 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.715842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.715899 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.715916 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:35.716129 (+ 213us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:35.716132 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.716185 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.716239 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260924 15:23:35.718494 12384 raft_consensus.cc:1261] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Refusing update from remote peer 667afa2810894b09818663b474f3a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:35.718525 12242 raft_consensus.cc:1261] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Refusing update from remote peer 667afa2810894b09818663b474f3a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:35.718719 12478 consensus_queue.cc:1035] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:35.718794 12466 consensus_queue.cc:1035] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:35.720642 12508 mvcc.cc:205] Tried to move back new op lower bound from 7332918950781583360 to 7332918949344894976. Current Snapshot: MvccSnapshot[applied={T|T < 7332918950781583360}]
I20260924 15:23:35.727766 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.724904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:39183 (local address 127.4.78.65:43685)
0924 15:23:35.724950 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.724952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.725076 (+ 124us) server_negotiation.cc:407] Connection header received
0924 15:23:35.725143 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.725144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.725157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.725222 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:35.725360 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.726131 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.727032 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.727312 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.727390 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.727527 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:35.727562 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:35.727630 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.727678 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.727699 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:23:35.727787 12521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.724869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43685 (local address 127.4.78.67:39183)
0924 15:23:35.725014 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:35.725024 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:35.725082 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:35.725234 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:35.725234 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:35.725236 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:23:35.725295 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.725296 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.726208 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.726208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.726979 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:35.726981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.727392 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.727392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:35.727438 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.727450 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:23:35.727638 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:23:35.727640 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:23:35.727674 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 15:23:35.727707 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260924 15:23:35.734408 11942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:35.734494 11942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:35.735083 11942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: Sending DeleteTablet for 3 replicas of tablet 3e38c9c6cd654c88acce08ec188083b1
I20260924 15:23:35.735337 12080 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e38c9c6cd654c88acce08ec188083b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:55910
I20260924 15:23:35.735422 12222 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e38c9c6cd654c88acce08ec188083b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:34216
I20260924 15:23:35.735612 12364 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e38c9c6cd654c88acce08ec188083b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:54680
I20260924 15:23:35.735805 12525 tablet_replica.cc:330] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: stopping tablet replica
I20260924 15:23:35.735805 12523 tablet_replica.cc:330] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: stopping tablet replica
I20260924 15:23:35.735862 12523 raft_consensus.cc:2229] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:35.735898 12525 raft_consensus.cc:2229] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:35.735965 12523 raft_consensus.cc:2258] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.735975 12525 raft_consensus.cc:2258] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.736238 12523 ts_tablet_manager.cc:1905] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.738013 12523 ts_tablet_manager.cc:1918] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:35.738291 12523 log.cc:1196] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/3e38c9c6cd654c88acce08ec188083b1
I20260924 15:23:35.738595 12523 ts_tablet_manager.cc:1939] T 3e38c9c6cd654c88acce08ec188083b1 P 667afa2810894b09818663b474f3a90e: Deleting consensus metadata
I20260924 15:23:35.738607 12525 ts_tablet_manager.cc:1905] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.738909 11930 catalog_manager.cc:4841] TS 667afa2810894b09818663b474f3a90e (127.4.78.65:43685): tablet 3e38c9c6cd654c88acce08ec188083b1 (table test-table [id=4c7061132c074a6faed35bdb86334c1f]) successfully deleted
I20260924 15:23:35.738953 12524 tablet_replica.cc:330] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: stopping tablet replica
I20260924 15:23:35.739008 12524 raft_consensus.cc:2229] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:35.739094 12524 raft_consensus.cc:2258] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.739337 12524 ts_tablet_manager.cc:1905] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.740422 12525 ts_tablet_manager.cc:1918] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:35.740491 12525 log.cc:1196] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/3e38c9c6cd654c88acce08ec188083b1
I20260924 15:23:35.740742 12524 ts_tablet_manager.cc:1918] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:35.740747 12525 ts_tablet_manager.cc:1939] T 3e38c9c6cd654c88acce08ec188083b1 P b32072fd474c445d82eed389283e2a2e: Deleting consensus metadata
I20260924 15:23:35.740814 12524 log.cc:1196] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/3e38c9c6cd654c88acce08ec188083b1
I20260924 15:23:35.741084 11930 catalog_manager.cc:4841] TS b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149): tablet 3e38c9c6cd654c88acce08ec188083b1 (table test-table [id=4c7061132c074a6faed35bdb86334c1f]) successfully deleted
I20260924 15:23:35.741127 12524 ts_tablet_manager.cc:1939] T 3e38c9c6cd654c88acce08ec188083b1 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting consensus metadata
I20260924 15:23:35.741415 11930 catalog_manager.cc:4841] TS 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165): tablet 3e38c9c6cd654c88acce08ec188083b1 (table test-table [id=4c7061132c074a6faed35bdb86334c1f]) successfully deleted
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:23:35.762948 11992 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:35.756736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.4.78.126:37759)
0924 15:23:35.756808 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.756811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.756937 (+ 126us) server_negotiation.cc:407] Connection header received
0924 15:23:35.757007 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.757008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.757018 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.757067 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:35.757798 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.758361 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.759482 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.759661 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.759712 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.762849 (+ 3137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42720: BlockingRecv error: recv got EOF from 127.0.0.1:42720 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 15:23:35 dist-test-slave-l766 krb5kdc[11898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM
I20260924 15:23:35.770845 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.763859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42730 (local address 127.4.78.126:37759)
0924 15:23:35.763898 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.763899 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.763988 (+ 89us) server_negotiation.cc:407] Connection header received
0924 15:23:35.764063 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.764064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.764073 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.764125 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:35.764591 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.765114 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.765793 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.766020 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.766094 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.768268 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:35.768269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:35.768270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:35.768277 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:35.769657 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:35.770049 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:35.770050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:35.770050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:35.770119 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:35.770333 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:35.770334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:35.770334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:35.770456 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:35.770567 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.770657 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.770734 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:23:35.772477 11942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:35.772651 11942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:35.774011 12080 tablet_service.cc:1467] Processing CreateTablet for tablet a77079fbd6cf439bba3d96705412a022 (DEFAULT_TABLE table=test-table [id=a0dfce7ac3134af99e0dc052fa19aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.774067 12364 tablet_service.cc:1467] Processing CreateTablet for tablet a77079fbd6cf439bba3d96705412a022 (DEFAULT_TABLE table=test-table [id=a0dfce7ac3134af99e0dc052fa19aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.774094 12222 tablet_service.cc:1467] Processing CreateTablet for tablet a77079fbd6cf439bba3d96705412a022 (DEFAULT_TABLE table=test-table [id=a0dfce7ac3134af99e0dc052fa19aa5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.774159 12080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77079fbd6cf439bba3d96705412a022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.774200 12222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77079fbd6cf439bba3d96705412a022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.774201 12364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a77079fbd6cf439bba3d96705412a022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.775552 12458 tablet_bootstrap.cc:492] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: Bootstrap starting.
I20260924 15:23:35.775563 12461 tablet_bootstrap.cc:492] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: Bootstrap starting.
I20260924 15:23:35.775959 12458 tablet_bootstrap.cc:654] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.776012 12461 tablet_bootstrap.cc:654] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.776376 12460 tablet_bootstrap.cc:492] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: Bootstrap starting.
I20260924 15:23:35.776584 12458 tablet_bootstrap.cc:492] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: No bootstrap required, opened a new log
I20260924 15:23:35.776618 12458 ts_tablet_manager.cc:1397] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:35.776684 12460 tablet_bootstrap.cc:654] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.776736 12458 raft_consensus.cc:348] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.776783 12458 raft_consensus.cc:374] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.776796 12458 raft_consensus.cc:729] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32072fd474c445d82eed389283e2a2e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.776822 12458 consensus_queue.cc:260] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.776917 12458 ts_tablet_manager.cc:1428] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:35.777660 12460 tablet_bootstrap.cc:492] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: No bootstrap required, opened a new log
I20260924 15:23:35.777715 12460 ts_tablet_manager.cc:1397] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:35.777837 12460 raft_consensus.cc:348] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.777906 12460 raft_consensus.cc:374] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.777926 12460 raft_consensus.cc:729] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667afa2810894b09818663b474f3a90e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.777962 12460 consensus_queue.cc:260] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.778056 12460 ts_tablet_manager.cc:1428] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:35.778316 12461 tablet_bootstrap.cc:492] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: No bootstrap required, opened a new log
I20260924 15:23:35.778362 12461 ts_tablet_manager.cc:1397] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 15:23:35.778503 12461 raft_consensus.cc:348] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.778564 12461 raft_consensus.cc:374] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.778584 12461 raft_consensus.cc:729] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c520f5eb7c94c8fbacdac885f19d916, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.778630 12461 consensus_queue.cc:260] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.778707 12461 ts_tablet_manager.cc:1428] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:35.945969 12462 raft_consensus.cc:482] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:35.946085 12462 raft_consensus.cc:504] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.946286 12462 leader_election.cc:288] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 667afa2810894b09818663b474f3a90e (127.4.78.65:43685), 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165)
I20260924 15:23:35.946545 12101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77079fbd6cf439bba3d96705412a022" candidate_uuid: "b32072fd474c445d82eed389283e2a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667afa2810894b09818663b474f3a90e" is_pre_election: true
I20260924 15:23:35.946631 12101 raft_consensus.cc:2454] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32072fd474c445d82eed389283e2a2e in term 0.
I20260924 15:23:35.946569 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77079fbd6cf439bba3d96705412a022" candidate_uuid: "b32072fd474c445d82eed389283e2a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916" is_pre_election: true
I20260924 15:23:35.946712 12242 raft_consensus.cc:2454] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b32072fd474c445d82eed389283e2a2e in term 0.
I20260924 15:23:35.946834 12318 leader_election.cc:302] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, b32072fd474c445d82eed389283e2a2e; no voters:
I20260924 15:23:35.946933 12462 raft_consensus.cc:2781] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:35.946971 12462 raft_consensus.cc:482] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:35.946998 12462 raft_consensus.cc:3037] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.947651 12462 raft_consensus.cc:504] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.947783 12462 leader_election.cc:288] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [CANDIDATE]: Term 1 election: Requested vote from peers 667afa2810894b09818663b474f3a90e (127.4.78.65:43685), 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165)
I20260924 15:23:35.947962 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77079fbd6cf439bba3d96705412a022" candidate_uuid: "b32072fd474c445d82eed389283e2a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916"
I20260924 15:23:35.947943 12101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a77079fbd6cf439bba3d96705412a022" candidate_uuid: "b32072fd474c445d82eed389283e2a2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "667afa2810894b09818663b474f3a90e"
I20260924 15:23:35.948033 12242 raft_consensus.cc:3037] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.948033 12101 raft_consensus.cc:3037] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:35.948781 12242 raft_consensus.cc:2454] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32072fd474c445d82eed389283e2a2e in term 1.
I20260924 15:23:35.948781 12101 raft_consensus.cc:2454] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b32072fd474c445d82eed389283e2a2e in term 1.
I20260924 15:23:35.948954 12318 leader_election.cc:302] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, b32072fd474c445d82eed389283e2a2e; no voters:
I20260924 15:23:35.949126 12462 raft_consensus.cc:2781] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:35.949182 12462 raft_consensus.cc:686] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 1 LEADER]: Becoming Leader. State: Replica: b32072fd474c445d82eed389283e2a2e, State: Running, Role: LEADER
I20260924 15:23:35.949244 12462 consensus_queue.cc:237] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.949937 11943 catalog_manager.cc:5495] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e reported cstate change: term changed from 0 to 1, leader changed from <none> to b32072fd474c445d82eed389283e2a2e (127.4.78.67). New cstate: current_term: 1 leader_uuid: "b32072fd474c445d82eed389283e2a2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } health_report { overall_health: HEALTHY } } }
I20260924 15:23:35.960220 12504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:35.957879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.4.78.67:43149)
0924 15:23:35.957921 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:35.957924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:35.957932 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:35.957997 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:35.957998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:35.958012 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:35.958067 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:35.958194 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.958726 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.959515 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:35.959717 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:35.959807 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:35.959955 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:35.959996 (+ 41us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:35.960059 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:35.960122 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 15:23:35.960143 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:23:35.961004 12242 raft_consensus.cc:1261] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Refusing update from remote peer b32072fd474c445d82eed389283e2a2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:35.961037 12101 raft_consensus.cc:1261] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Refusing update from remote peer b32072fd474c445d82eed389283e2a2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:35.961216 12462 consensus_queue.cc:1035] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:35.961356 12512 consensus_queue.cc:1035] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:35.964604 11942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:35.964692 11942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:35.965253 11942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: Sending DeleteTablet for 3 replicas of tablet a77079fbd6cf439bba3d96705412a022
I20260924 15:23:35.965553 12080 tablet_service.cc:1514] Processing DeleteTablet for tablet a77079fbd6cf439bba3d96705412a022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:55910
I20260924 15:23:35.965605 12222 tablet_service.cc:1514] Processing DeleteTablet for tablet a77079fbd6cf439bba3d96705412a022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:34216
I20260924 15:23:35.965652 12523 tablet_replica.cc:330] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: stopping tablet replica
I20260924 15:23:35.965703 12523 raft_consensus.cc:2229] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:35.965705 12364 tablet_service.cc:1514] Processing DeleteTablet for tablet a77079fbd6cf439bba3d96705412a022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:35 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:54680
I20260924 15:23:35.965752 12523 raft_consensus.cc:2258] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.965792 12524 tablet_replica.cc:330] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: stopping tablet replica
I20260924 15:23:35.965783 12525 tablet_replica.cc:330] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: stopping tablet replica
I20260924 15:23:35.965862 12524 raft_consensus.cc:2229] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:35.965867 12525 raft_consensus.cc:2229] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:35.965898 12524 raft_consensus.cc:2258] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.965952 12525 raft_consensus.cc:2258] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:35.966138 11942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:35.966284 11942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:35.966281 12525 ts_tablet_manager.cc:1905] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.966684 12523 ts_tablet_manager.cc:1905] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.967726 12525 ts_tablet_manager.cc:1918] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:35.967785 12525 log.cc:1196] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/a77079fbd6cf439bba3d96705412a022
I20260924 15:23:35.968019 12525 ts_tablet_manager.cc:1939] T a77079fbd6cf439bba3d96705412a022 P b32072fd474c445d82eed389283e2a2e: Deleting consensus metadata
I20260924 15:23:35.968245 12523 ts_tablet_manager.cc:1918] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:35.968293 12523 log.cc:1196] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/a77079fbd6cf439bba3d96705412a022
I20260924 15:23:35.968340 11930 catalog_manager.cc:4841] TS b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149): tablet a77079fbd6cf439bba3d96705412a022 (table test-table [id=a0dfce7ac3134af99e0dc052fa19aa5d]) successfully deleted
I20260924 15:23:35.968493 12523 ts_tablet_manager.cc:1939] T a77079fbd6cf439bba3d96705412a022 P 667afa2810894b09818663b474f3a90e: Deleting consensus metadata
I20260924 15:23:35.968698 12222 tablet_service.cc:1467] Processing CreateTablet for tablet 02b98c28493e49ecb1d484742c45e3cd (DEFAULT_TABLE table=test-table [id=0588562648ad4163882d7d2dcba8a635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.968726 12364 tablet_service.cc:1467] Processing CreateTablet for tablet 02b98c28493e49ecb1d484742c45e3cd (DEFAULT_TABLE table=test-table [id=0588562648ad4163882d7d2dcba8a635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.968806 12222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b98c28493e49ecb1d484742c45e3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.968843 12364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b98c28493e49ecb1d484742c45e3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.969944 12458 tablet_bootstrap.cc:492] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: Bootstrap starting.
I20260924 15:23:35.969926 12080 tablet_service.cc:1467] Processing CreateTablet for tablet 02b98c28493e49ecb1d484742c45e3cd (DEFAULT_TABLE table=test-table [id=0588562648ad4163882d7d2dcba8a635]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:35.970017 12080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b98c28493e49ecb1d484742c45e3cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:35.970178 11930 catalog_manager.cc:4841] TS 667afa2810894b09818663b474f3a90e (127.4.78.65:43685): tablet a77079fbd6cf439bba3d96705412a022 (table test-table [id=a0dfce7ac3134af99e0dc052fa19aa5d]) successfully deleted
I20260924 15:23:35.970243 12458 tablet_bootstrap.cc:654] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.970813 12458 tablet_bootstrap.cc:492] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: No bootstrap required, opened a new log
I20260924 15:23:35.970855 12458 ts_tablet_manager.cc:1397] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:23:35.970950 12458 raft_consensus.cc:348] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.971001 12458 raft_consensus.cc:374] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.971019 12458 raft_consensus.cc:729] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b32072fd474c445d82eed389283e2a2e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.971060 12458 consensus_queue.cc:260] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.971158 12458 ts_tablet_manager.cc:1428] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:35.971361 12460 tablet_bootstrap.cc:492] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e: Bootstrap starting.
I20260924 15:23:35.971619 12460 tablet_bootstrap.cc:654] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.968809 12524 ts_tablet_manager.cc:1905] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:35.972810 12460 tablet_bootstrap.cc:492] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e: No bootstrap required, opened a new log
I20260924 15:23:35.972854 12460 ts_tablet_manager.cc:1397] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260924 15:23:35.972882 12524 ts_tablet_manager.cc:1918] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:35.972927 12524 log.cc:1196] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/a77079fbd6cf439bba3d96705412a022
I20260924 15:23:35.972952 12460 raft_consensus.cc:348] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.973021 12460 raft_consensus.cc:374] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.973044 12460 raft_consensus.cc:729] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 667afa2810894b09818663b474f3a90e, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.973099 12460 consensus_queue.cc:260] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.973135 12524 ts_tablet_manager.cc:1939] T a77079fbd6cf439bba3d96705412a022 P 9c520f5eb7c94c8fbacdac885f19d916: Deleting consensus metadata
I20260924 15:23:35.973199 12460 ts_tablet_manager.cc:1428] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:35.973448 11930 catalog_manager.cc:4841] TS 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165): tablet a77079fbd6cf439bba3d96705412a022 (table test-table [id=a0dfce7ac3134af99e0dc052fa19aa5d]) successfully deleted
I20260924 15:23:35.974442 12461 tablet_bootstrap.cc:492] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: Bootstrap starting.
I20260924 15:23:35.974788 12461 tablet_bootstrap.cc:654] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:35.975381 12461 tablet_bootstrap.cc:492] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: No bootstrap required, opened a new log
I20260924 15:23:35.975440 12461 ts_tablet_manager.cc:1397] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:35.975608 12461 raft_consensus.cc:348] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.975682 12461 raft_consensus.cc:374] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:35.975706 12461 raft_consensus.cc:729] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c520f5eb7c94c8fbacdac885f19d916, State: Initialized, Role: FOLLOWER
I20260924 15:23:35.975754 12461 consensus_queue.cc:260] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:35.975844 12461 ts_tablet_manager.cc:1428] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:36.229916 12466 raft_consensus.cc:482] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:36.230005 12466 raft_consensus.cc:504] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:36.230176 12466 leader_election.cc:288] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165), b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149)
I20260924 15:23:36.230470 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b98c28493e49ecb1d484742c45e3cd" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916" is_pre_election: true
I20260924 15:23:36.230470 12384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b98c28493e49ecb1d484742c45e3cd" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32072fd474c445d82eed389283e2a2e" is_pre_election: true
I20260924 15:23:36.230578 12384 raft_consensus.cc:2454] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 0.
I20260924 15:23:36.230578 12242 raft_consensus.cc:2454] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 0.
I20260924 15:23:36.230760 12034 leader_election.cc:302] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, 9c520f5eb7c94c8fbacdac885f19d916; no voters:
I20260924 15:23:36.230873 12466 raft_consensus.cc:2781] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:36.230926 12466 raft_consensus.cc:482] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:36.230957 12466 raft_consensus.cc:3037] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:36.231778 12466 raft_consensus.cc:504] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:36.231905 12466 leader_election.cc:288] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 election: Requested vote from peers 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165), b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149)
I20260924 15:23:36.232052 12242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b98c28493e49ecb1d484742c45e3cd" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c520f5eb7c94c8fbacdac885f19d916"
I20260924 15:23:36.232112 12384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b98c28493e49ecb1d484742c45e3cd" candidate_uuid: "667afa2810894b09818663b474f3a90e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b32072fd474c445d82eed389283e2a2e"
I20260924 15:23:36.232136 12242 raft_consensus.cc:3037] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:36.232191 12384 raft_consensus.cc:3037] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:36.232796 12384 raft_consensus.cc:2454] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 1.
I20260924 15:23:36.232863 12242 raft_consensus.cc:2454] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 667afa2810894b09818663b474f3a90e in term 1.
I20260924 15:23:36.233011 12034 leader_election.cc:302] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 667afa2810894b09818663b474f3a90e, b32072fd474c445d82eed389283e2a2e; no voters:
I20260924 15:23:36.233135 12466 raft_consensus.cc:2781] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:36.233253 12466 raft_consensus.cc:686] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 1 LEADER]: Becoming Leader. State: Replica: 667afa2810894b09818663b474f3a90e, State: Running, Role: LEADER
I20260924 15:23:36.233328 12466 consensus_queue.cc:237] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } }
I20260924 15:23:36.233904 11942 catalog_manager.cc:5495] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e reported cstate change: term changed from 0 to 1, leader changed from <none> to 667afa2810894b09818663b474f3a90e (127.4.78.65). New cstate: current_term: 1 leader_uuid: "667afa2810894b09818663b474f3a90e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "667afa2810894b09818663b474f3a90e" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 } health_report { overall_health: UNKNOWN } } }
I20260924 15:23:36.261289 12551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:36.259236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55932 (local address 127.4.78.65:43685)
0924 15:23:36.259338 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:36.259340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:36.259347 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:23:36.259366 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:36.259367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:36.259375 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:36.259428 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:36.259517 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:36.259994 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:36.260692 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:36.260891 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:36.260940 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:36.261041 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:36.261065 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:36.261107 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:36.261175 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:23:36.261189 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":61,"threads_started":1}
I20260924 15:23:36.263517 12242 raft_consensus.cc:1261] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Refusing update from remote peer 667afa2810894b09818663b474f3a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:36.263530 12384 raft_consensus.cc:1261] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Refusing update from remote peer 667afa2810894b09818663b474f3a90e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:36.263705 12549 consensus_queue.cc:1035] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b32072fd474c445d82eed389283e2a2e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:36.263818 12466 consensus_queue.cc:1035] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c520f5eb7c94c8fbacdac885f19d916" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:36.273765 11942 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:36.273892 11942 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42730:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:36.274449 11942 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0871a092153640dd84db1a9df7c60cc6: Sending DeleteTablet for 3 replicas of tablet 02b98c28493e49ecb1d484742c45e3cd
I20260924 15:23:36.274698 12222 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b98c28493e49ecb1d484742c45e3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:36 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:34216
I20260924 15:23:36.274721 12080 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b98c28493e49ecb1d484742c45e3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:36 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:55910
I20260924 15:23:36.274806 12524 tablet_replica.cc:330] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: stopping tablet replica
I20260924 15:23:36.274828 12523 tablet_replica.cc:330] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e: stopping tablet replica
I20260924 15:23:36.274847 12364 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b98c28493e49ecb1d484742c45e3cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:36 UTC) from {username='oryx', principal='oryx/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:54680
I20260924 15:23:36.274874 12524 raft_consensus.cc:2229] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:36.274878 12523 raft_consensus.cc:2229] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:36.274912 12524 raft_consensus.cc:2258] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:36.274926 12523 raft_consensus.cc:2258] T 02b98c28493e49ecb1d484742c45e3cd P 667afa2810894b09818663b474f3a90e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:36.274945 12525 tablet_replica.cc:330] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: stopping tablet replica
I20260924 15:23:36.274999 12525 raft_consensus.cc:2229] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:36.275032 12525 raft_consensus.cc:2258] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:36.275264 12524 ts_tablet_manager.cc:1905] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:36.275278 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 12015
I20260924 15:23:36.275336 12525 ts_tablet_manager.cc:1905] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:36.276957 12524 ts_tablet_manager.cc:1918] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:36.277034 12524 log.cc:1196] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/02b98c28493e49ecb1d484742c45e3cd
I20260924 15:23:36.277159 12525 ts_tablet_manager.cc:1918] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:23:36.277225 12525 log.cc:1196] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/02b98c28493e49ecb1d484742c45e3cd
I20260924 15:23:36.277281 12524 ts_tablet_manager.cc:1939] T 02b98c28493e49ecb1d484742c45e3cd P 9c520f5eb7c94c8fbacdac885f19d916: Deleting consensus metadata
I20260924 15:23:36.277449 12525 ts_tablet_manager.cc:1939] T 02b98c28493e49ecb1d484742c45e3cd P b32072fd474c445d82eed389283e2a2e: Deleting consensus metadata
I20260924 15:23:36.277842 11930 catalog_manager.cc:4841] TS 9c520f5eb7c94c8fbacdac885f19d916 (127.4.78.66:43165): tablet 02b98c28493e49ecb1d484742c45e3cd (table test-table [id=0588562648ad4163882d7d2dcba8a635]) successfully deleted
I20260924 15:23:36.278111 11930 catalog_manager.cc:4841] TS b32072fd474c445d82eed389283e2a2e (127.4.78.67:43149): tablet 02b98c28493e49ecb1d484742c45e3cd (table test-table [id=0588562648ad4163882d7d2dcba8a635]) successfully deleted
W20260924 15:23:36.283141 11930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:43685 (error 108)
I20260924 15:23:36.283299 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 12156
I20260924 15:23:36.286167 12559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:36.285966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43685 (local address 127.0.0.1:55938)
0924 15:23:36.286066 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:36.286121 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:43685: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":40,"threads_started":1}
W20260924 15:23:36.286311 11930 catalog_manager.cc:4584] TS 667afa2810894b09818663b474f3a90e (127.4.78.65:43685): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 02b98c28493e49ecb1d484742c45e3cd: Network error: Client connection negotiation failed: client connection to 127.4.78.65:43685: connect: Connection refused (error 111)
I20260924 15:23:36.288491 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 12298
I20260924 15:23:36.294400 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 11911
2026-09-24T15:23:36Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3099 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:36 dist-test-slave-l766 krb5kdc[12562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:36 dist-test-slave-l766 krb5kdc[12562](info): set up 2 sockets
Sep 24 15:23:36 dist-test-slave-l766 krb5kdc[12562](info): commencing operation
krb5kdc: starting...
W20260924 15:23:37.341284 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:37 dist-test-slave-l766 krb5kdc[12562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:37.454149 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:40361
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:37.533372 12575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:37.533571 12575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:37.533593 12575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:37.535025 12575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:37.535080 12575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:37.535096 12575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:37.535166 12575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:37.535188 12575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:37.536732 12575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:40361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:37.536983 12575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:37.537223 12575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:37.541899 12583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:37.542003 12575 server_base.cc:1034] running on GCE node
W20260924 15:23:37.541903 12580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:37.541937 12581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:37.542330 12575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:37.542661 12575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:37.543823 12575 hybrid_clock.cc:648] HybridClock initialized: now 1790263417543800 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T15:23:37Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1255 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:37 dist-test-slave-l766 krb5kdc[12588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:37 dist-test-slave-l766 krb5kdc[12588](info): set up 2 sockets
Sep 24 15:23:37 dist-test-slave-l766 krb5kdc[12588](info): commencing operation
krb5kdc: starting...
W20260924 15:23:38.593300 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:38 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:38.705152 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39369
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:38.783839 12601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:38.784075 12601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:38.784108 12601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:38.785190 12601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:38.785254 12601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:38.785276 12601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:38.785295 12601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:38.785324 12601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:38.786803 12601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:38.787066 12601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:38.787286 12601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:38.791953 12609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:38.791929 12607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:38.791998 12601 server_base.cc:1034] running on GCE node
W20260924 15:23:38.791963 12606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:38.792357 12601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:38.792675 12601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:38.793870 12601 hybrid_clock.cc:648] HybridClock initialized: now 1790263418793854 us; error 26 us; skew 500 ppm
Sep 24 15:23:38 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263418, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:38.796144 12601 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:38.796583 12601 webserver.cc:460] Webserver started at http://127.4.78.126:40679/ using document root <none> and password file <none>
I20260924 15:23:38.796808 12601 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:38.796872 12601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:38.796999 12601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:38.798094 12601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "1ab26934e6a94cf7a7e37031c2303b6b"
format_stamp: "Formatted at 2026-09-24 15:23:38 on dist-test-slave-l766"
server_key: "d5c7d57973c03acea614c816893feafb"
server_key_iv: "1b99b9899b18df0027ed82ec55064cf9"
server_key_version: "encryptionkey@0"
I20260924 15:23:38.798452 12601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "1ab26934e6a94cf7a7e37031c2303b6b"
format_stamp: "Formatted at 2026-09-24 15:23:38 on dist-test-slave-l766"
server_key: "d5c7d57973c03acea614c816893feafb"
server_key_iv: "1b99b9899b18df0027ed82ec55064cf9"
server_key_version: "encryptionkey@0"
I20260924 15:23:38.799759 12601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:38.800519 12616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:38.800693 12601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:38.800791 12601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "1ab26934e6a94cf7a7e37031c2303b6b"
format_stamp: "Formatted at 2026-09-24 15:23:38 on dist-test-slave-l766"
server_key: "d5c7d57973c03acea614c816893feafb"
server_key_iv: "1b99b9899b18df0027ed82ec55064cf9"
server_key_version: "encryptionkey@0"
I20260924 15:23:38.800871 12601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:38.811595 12601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:38.813246 12601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:38.813400 12601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:38.817341 12601 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:39369
I20260924 15:23:38.817373 12668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:39369 every 8 connection(s)
I20260924 15:23:38.817736 12601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:38.818586 12669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:38.819468 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 12601
I20260924 15:23:38.819578 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:38.819775 4409 external_mini_cluster.cc:1428] Setting key ffedff5359ea10e48c3ee23ca315c0d1
I20260924 15:23:38.821027 12669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Bootstrap starting.
I20260924 15:23:38.821595 12669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:38.821954 12669 log.cc:824] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:38.822613 12669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: No bootstrap required, opened a new log
I20260924 15:23:38.824352 12669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } }
I20260924 15:23:38.824507 12669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:38.824571 12669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab26934e6a94cf7a7e37031c2303b6b, State: Initialized, Role: FOLLOWER
I20260924 15:23:38.824710 12669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } }
I20260924 15:23:38.824810 12669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:38.824855 12669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:38.824901 12669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:38.825462 12669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } }
I20260924 15:23:38.825548 12669 leader_election.cc:302] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ab26934e6a94cf7a7e37031c2303b6b; no voters:
I20260924 15:23:38.825717 12669 leader_election.cc:288] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:38.825778 12674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:38.825994 12669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:38.826001 12674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 1 LEADER]: Becoming Leader. State: Replica: 1ab26934e6a94cf7a7e37031c2303b6b, State: Running, Role: LEADER
I20260924 15:23:38.826253 12674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } }
Sep 24 15:23:38 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:38.826752 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ab26934e6a94cf7a7e37031c2303b6b. Latest consensus state: current_term: 1 leader_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } } }
I20260924 15:23:38.826869 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:38.827119 12675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } } }
I20260924 15:23:38.827204 12675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:38.827548 12683 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:38.828339 12683 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:38.830036 12683 catalog_manager.cc:1347] Generated new cluster ID: 65e5cdbc640f4d63a8ea7616649e51ce
I20260924 15:23:38.830096 12683 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:38.831249 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:38.820303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41824 (local address 127.4.78.126:39369)
0924 15:23:38.820477 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:38.820484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:38.820662 (+ 178us) server_negotiation.cc:407] Connection header received
0924 15:23:38.821170 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:38.821180 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:38.821239 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:38.821384 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:38.822149 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:38.823096 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:38.823879 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:38.824366 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:38.824485 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:38.827336 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:38.827339 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:38.827340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:38.827360 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:38.829288 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:38.829704 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:38.829707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:38.829707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:38.829803 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:38.830039 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:38.830040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:38.830040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:38.830356 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:38.830474 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:38.830914 (+ 440us) server_negotiation.cc:299] Negotiation successful
0924 15:23:38.831033 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260924 15:23:38.851894 12683 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:38.852599 12683 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:38.864979 12683 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Generated new TSK 0
I20260924 15:23:38.865159 12683 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:38.890153 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39369
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:38.968854 12695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:38.969054 12695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:38.969077 12695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:38.970257 12695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:38.970299 12695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:38.970324 12695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:38.971745 12695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:38.971971 12695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:38.972185 12695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:38.976962 12703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:38.976961 12700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:38.977089 12695 server_base.cc:1034] running on GCE node
W20260924 15:23:38.976970 12701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:38.977481 12695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:38.977667 12695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:38.978832 12695 hybrid_clock.cc:648] HybridClock initialized: now 1790263418978813 us; error 34 us; skew 500 ppm
Sep 24 15:23:38 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263418, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:38.981207 12695 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:38.981603 12695 webserver.cc:460] Webserver started at http://127.4.78.65:37429/ using document root <none> and password file <none>
I20260924 15:23:38.981930 12695 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:38.981998 12695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:38.982131 12695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:38.983134 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "8839c6a7e00943c9933c565b4dbd6089"
format_stamp: "Formatted at 2026-09-24 15:23:38 on dist-test-slave-l766"
server_key: "51fc4291ce17cb3e61a29e64302c5143"
server_key_iv: "07f069ff2ecffcbcc3583106ecd45ced"
server_key_version: "encryptionkey@0"
I20260924 15:23:38.983491 12695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "8839c6a7e00943c9933c565b4dbd6089"
format_stamp: "Formatted at 2026-09-24 15:23:38 on dist-test-slave-l766"
server_key: "51fc4291ce17cb3e61a29e64302c5143"
server_key_iv: "07f069ff2ecffcbcc3583106ecd45ced"
server_key_version: "encryptionkey@0"
I20260924 15:23:38.984875 12695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:38.985718 12710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:38.985935 12695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:38.986034 12695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "8839c6a7e00943c9933c565b4dbd6089"
format_stamp: "Formatted at 2026-09-24 15:23:38 on dist-test-slave-l766"
server_key: "51fc4291ce17cb3e61a29e64302c5143"
server_key_iv: "07f069ff2ecffcbcc3583106ecd45ced"
server_key_version: "encryptionkey@0"
I20260924 15:23:38.986116 12695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:39.002965 12695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:39.004916 12695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:39.005081 12695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:39.005311 12695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:39.005621 12695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:39.005676 12695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.005709 12695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:39.005738 12695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.011497 12695 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:38145
I20260924 15:23:39.011545 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:38145 every 8 connection(s)
I20260924 15:23:39.011842 12695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:39.014743 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 12695
I20260924 15:23:39.014881 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:39.015074 4409 external_mini_cluster.cc:1428] Setting key 7bd668bbe43de1144b88b44e1a067b69
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263418, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:39.022035 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.012560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:57393 (local address 127.4.78.126:39369)
0924 15:23:39.012619 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.012621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.012834 (+ 213us) server_negotiation.cc:407] Connection header received
0924 15:23:39.013282 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.013283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.013292 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.013366 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.014336 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.014852 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.016126 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.016471 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.016545 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.019011 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:39.019013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:39.019013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:39.019019 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:39.020489 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:39.021042 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:39.021042 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:39.021043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:39.021094 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:39.021367 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:39.021368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:39.021368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:39.021465 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:39.021515 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:39.021889 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 15:23:39.021968 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:23:39.022270 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.012542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:57393)
0924 15:23:39.012700 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.012740 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.013206 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.013416 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.013420 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.014067 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.014252 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.014257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.014989 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.014991 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.016037 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.016040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.016556 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.016556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.016647 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.017179 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.017189 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.018922 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:39.020575 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:39.020583 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:39.020585 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:39.020957 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:39.021159 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:39.021160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:39.021160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:39.021300 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:39.021533 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:39.021540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:39.021714 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 15:23:39.021851 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 15:23:39.022021 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260924 15:23:39.022858 12825 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39369
I20260924 15:23:39.022959 12825 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:39.023149 12825 heartbeater.cc:511] Master 127.4.78.126:39369 requested a full tablet report, sending...
I20260924 15:23:39.023617 12633 ts_manager.cc:194] Registered new tserver with Master: 8839c6a7e00943c9933c565b4dbd6089 (127.4.78.65:38145)
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
I20260924 15:23:39.024245 12633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:57393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:39.038829 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39369
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:39.117636 12830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:39.117865 12830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:39.117901 12830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:39.119010 12830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:39.119048 12830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:39.119072 12830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:39.120455 12830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:39.120697 12830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:39.120910 12830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:39.125626 12835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:39.125627 12836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.125733 12830 server_base.cc:1034] running on GCE node
W20260924 15:23:39.125707 12838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.126093 12830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:39.126372 12830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:39.127522 12830 hybrid_clock.cc:648] HybridClock initialized: now 1790263419127509 us; error 28 us; skew 500 ppm
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263419, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:39.130038 12830 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:39.130457 12830 webserver.cc:460] Webserver started at http://127.4.78.66:39433/ using document root <none> and password file <none>
I20260924 15:23:39.130673 12830 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:39.130718 12830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:39.130872 12830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:39.131909 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "f95535d70bbd4e4c95c14e3c161ba27f"
format_stamp: "Formatted at 2026-09-24 15:23:39 on dist-test-slave-l766"
server_key: "788999037fe98a50eb7555e032ffeb28"
server_key_iv: "935e4229f85069a58d756de780cc6164"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.132254 12830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "f95535d70bbd4e4c95c14e3c161ba27f"
format_stamp: "Formatted at 2026-09-24 15:23:39 on dist-test-slave-l766"
server_key: "788999037fe98a50eb7555e032ffeb28"
server_key_iv: "935e4229f85069a58d756de780cc6164"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.133549 12830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 15:23:39.134244 12845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.134398 12830 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.134506 12830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "f95535d70bbd4e4c95c14e3c161ba27f"
format_stamp: "Formatted at 2026-09-24 15:23:39 on dist-test-slave-l766"
server_key: "788999037fe98a50eb7555e032ffeb28"
server_key_iv: "935e4229f85069a58d756de780cc6164"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.134577 12830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:39.153029 12830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:39.154860 12830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:39.155025 12830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:39.155246 12830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:39.155548 12830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:39.155603 12830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.155644 12830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:39.155673 12830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.161315 12830 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:38639
I20260924 15:23:39.161367 12959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:38639 every 8 connection(s)
I20260924 15:23:39.161654 12830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:39.163050 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 12830
I20260924 15:23:39.163170 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:39.163386 4409 external_mini_cluster.cc:1428] Setting key 52a3b32955c3a07ac15f7fca18d5c102
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263419, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:39.172034 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.162398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:45677 (local address 127.4.78.126:39369)
0924 15:23:39.162459 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.162462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.162632 (+ 170us) server_negotiation.cc:407] Connection header received
0924 15:23:39.163263 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.163265 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.163286 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.163348 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.164613 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.165095 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.166275 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.166603 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.166682 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.169064 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:39.169065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:39.169066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:39.169074 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:39.170690 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:39.171062 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:39.171064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:39.171065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:39.171121 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:39.171395 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:39.171397 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:39.171397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:39.171540 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:39.171623 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:39.171844 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 15:23:39.171928 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:23:39.172065 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.162368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:45677)
0924 15:23:39.162521 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.162555 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.163159 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.163418 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.163422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.164253 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.164517 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.164523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.165181 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.165182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.166191 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.166194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.166695 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.166695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.166790 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.167267 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.167275 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.168952 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:39.170769 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:39.170775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:39.170777 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:39.170983 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:39.171224 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:39.171225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:39.171225 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:39.171322 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:39.171610 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:39.171614 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:39.171719 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 15:23:39.171796 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:23:39.171900 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
I20260924 15:23:39.172580 12960 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39369
I20260924 15:23:39.172696 12960 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:39.172880 12960 heartbeater.cc:511] Master 127.4.78.126:39369 requested a full tablet report, sending...
I20260924 15:23:39.173202 12633 ts_manager.cc:194] Registered new tserver with Master: f95535d70bbd4e4c95c14e3c161ba27f (127.4.78.66:38639)
I20260924 15:23:39.173682 12633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:45677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:39.188839 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39369
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:39.271538 12965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:39.271723 12965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:39.271745 12965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:39.272872 12965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:39.272908 12965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:39.272934 12965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:39.274335 12965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:39.274513 12965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:39.274773 12965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:39.280042 12973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.280160 12965 server_base.cc:1034] running on GCE node
W20260924 15:23:39.280115 12970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:39.280050 12971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.280522 12965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:39.280790 12965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:39.281927 12965 hybrid_clock.cc:648] HybridClock initialized: now 1790263419281918 us; error 29 us; skew 500 ppm
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263419, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:39.284281 12965 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:39.284679 12965 webserver.cc:460] Webserver started at http://127.4.78.67:46031/ using document root <none> and password file <none>
I20260924 15:23:39.284924 12965 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:39.284991 12965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:39.285147 12965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:39.286226 12965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "91a1faf599ef476285e336a395f360ee"
format_stamp: "Formatted at 2026-09-24 15:23:39 on dist-test-slave-l766"
server_key: "99df3f95776243a233a83659dc8d7a90"
server_key_iv: "bb0a1ab8304af4db4059ce8c9a1ad52d"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.286576 12965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "91a1faf599ef476285e336a395f360ee"
format_stamp: "Formatted at 2026-09-24 15:23:39 on dist-test-slave-l766"
server_key: "99df3f95776243a233a83659dc8d7a90"
server_key_iv: "bb0a1ab8304af4db4059ce8c9a1ad52d"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.287890 12965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:39.288591 12980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.288758 12965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:39.288856 12965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "91a1faf599ef476285e336a395f360ee"
format_stamp: "Formatted at 2026-09-24 15:23:39 on dist-test-slave-l766"
server_key: "99df3f95776243a233a83659dc8d7a90"
server_key_iv: "bb0a1ab8304af4db4059ce8c9a1ad52d"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.288932 12965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:39.296672 12965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:39.298420 12965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:39.298578 12965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:39.298786 12965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:39.299084 12965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:39.299139 12965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.299176 12965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:39.299204 12965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.305127 12965 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:39081
I20260924 15:23:39.305234 13093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:39081 every 8 connection(s)
I20260924 15:23:39.305537 12965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263419, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:39.313342 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 12965
I20260924 15:23:39.313473 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:39.313676 4409 external_mini_cluster.cc:1428] Setting key b3f515bf5d48698819821c73f6a750ba
I20260924 15:23:39.314693 12672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.306304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:54671 (local address 127.4.78.126:39369)
0924 15:23:39.306344 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.306346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.306554 (+ 208us) server_negotiation.cc:407] Connection header received
0924 15:23:39.307078 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.307079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.307092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.307157 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.308080 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.308611 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.309642 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.309907 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.309991 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.311924 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:39.311925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:39.311925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:39.311932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:39.313067 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:39.313577 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:39.313578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:39.313578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:39.313628 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:39.313934 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:39.313935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:39.313936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:39.314091 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:39.314162 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:39.314505 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 15:23:39.314588 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:23:39.314872 13096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.306282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:54671)
0924 15:23:39.306434 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.306467 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.306969 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.307225 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.307229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.307785 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.307969 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.307974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.308707 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.308709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.309526 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.309528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.310000 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.310001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.310129 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.310551 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.310559 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.311845 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:39.313133 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:39.313138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:39.313140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:39.313503 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:39.313695 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:39.313696 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:39.313696 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:39.313854 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:39.314168 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:39.314174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:39.314355 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 15:23:39.314465 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:23:39.314614 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260924 15:23:39.315325 13094 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39369
I20260924 15:23:39.315402 13094 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:39.315528 13094 heartbeater.cc:511] Master 127.4.78.126:39369 requested a full tablet report, sending...
I20260924 15:23:39.315838 12633 ts_manager.cc:194] Registered new tserver with Master: 91a1faf599ef476285e336a395f360ee (127.4.78.67:39081)
I20260924 15:23:39.316201 12633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:54671
I20260924 15:23:39.325884 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab.
Entry for principal oryx/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab.
WARNING: no policy specified for oryx/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab.
Entry for principal oryx/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab.
WARNING: no policy specified for oryx/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab.
Entry for principal oryx/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab.
I20260924 15:23:39.388345 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 12695
I20260924 15:23:39.393033 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 12830
I20260924 15:23:39.397441 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 12965
I20260924 15:23:39.401551 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 12601
I20260924 15:23:39.406634 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39369
--webserver_interface=127.4.78.126
--webserver_port=40679
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:39.485785 13103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:39.486017 13103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:39.486038 13103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:39.487246 13103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:39.487288 13103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:39.487303 13103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:39.487315 13103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:39.487365 13103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:39.488826 13103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=40679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:39.489079 13103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:39.489318 13103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:39.493925 13109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:39.493939 13108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:39.493896 13111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.494026 13103 server_base.cc:1034] running on GCE node
I20260924 15:23:39.494324 13103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:39.494589 13103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:39.495759 13103 hybrid_clock.cc:648] HybridClock initialized: now 1790263419495736 us; error 34 us; skew 500 ppm
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263419, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:39.498109 13103 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:39.498522 13103 webserver.cc:460] Webserver started at http://127.4.78.126:40679/ using document root <none> and password file <none>
I20260924 15:23:39.498741 13103 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:39.498823 13103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:39.500108 13103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 15:23:39.500794 13118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.500968 13103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:23:39.501072 13103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "1ab26934e6a94cf7a7e37031c2303b6b"
format_stamp: "Formatted at 2026-09-24 15:23:38 on dist-test-slave-l766"
server_key: "d5c7d57973c03acea614c816893feafb"
server_key_iv: "1b99b9899b18df0027ed82ec55064cf9"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.501394 13103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:39.513513 13103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:39.515851 13103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:39.516008 13103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:39.519802 13103 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:39369
I20260924 15:23:39.519838 13170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:39369 every 8 connection(s)
I20260924 15:23:39.520196 13103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:39.520965 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 13103
I20260924 15:23:39.521238 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:38145
--local_ip_for_outbound_sockets=127.4.78.65
--tserver_master_addrs=127.4.78.126:39369
--webserver_port=37429
--webserver_interface=127.4.78.65
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.78.65
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:23:39.521761 13171 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:23:39.522434 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Bootstrap starting.
I20260924 15:23:39.530310 13171 log.cc:824] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:39.532202 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:23:39.532510 13171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Bootstrap complete.
I20260924 15:23:39.534727 13171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } }
I20260924 15:23:39.534946 13171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab26934e6a94cf7a7e37031c2303b6b, State: Initialized, Role: FOLLOWER
I20260924 15:23:39.535112 13171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } }
I20260924 15:23:39.535175 13171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:39.535221 13171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:39.535266 13171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:23:39.536216 13171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } }
I20260924 15:23:39.536296 13171 leader_election.cc:302] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ab26934e6a94cf7a7e37031c2303b6b; no voters:
I20260924 15:23:39.536489 13171 leader_election.cc:288] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:23:39.536523 13175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:23:39.536650 13175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [term 2 LEADER]: Becoming Leader. State: Replica: 1ab26934e6a94cf7a7e37031c2303b6b, State: Running, Role: LEADER
I20260924 15:23:39.536747 13171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:39.536801 13175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } }
I20260924 15:23:39.537113 13175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ab26934e6a94cf7a7e37031c2303b6b. Latest consensus state: current_term: 2 leader_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } } }
I20260924 15:23:39.537211 13175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:39.537115 13177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab26934e6a94cf7a7e37031c2303b6b" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39369 } } }
I20260924 15:23:39.537599 13177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:39.538479 13191 catalog_manager.cc:1259] Loaded cluster ID: 65e5cdbc640f4d63a8ea7616649e51ce
I20260924 15:23:39.538570 13191 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: loading cluster ID for follower catalog manager: success
I20260924 15:23:39.539381 13191 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: acquiring CA information for follower catalog manager: success
I20260924 15:23:39.539685 13191 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:23:39.539767 13181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:39.539915 13181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:39.539996 13181 catalog_manager.cc:1259] Loaded cluster ID: 65e5cdbc640f4d63a8ea7616649e51ce
I20260924 15:23:39.540027 13181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:39.540199 13181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:39.540292 13181 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1ab26934e6a94cf7a7e37031c2303b6b: Loaded TSK: 0
I20260924 15:23:39.540436 13181 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:23:39.607214 13173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:39.607425 13173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:39.607461 13173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:39.608546 13173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:39.608605 13173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:39.608659 13173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:39.610133 13173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.65.keytab
--principal=oryx/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:38145
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=37429
--tserver_master_addrs=127.4.78.126:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:39.610383 13173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:39.610605 13173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:39.615381 13197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:39.615366 13200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:39.615366 13198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.615614 13173 server_base.cc:1034] running on GCE node
I20260924 15:23:39.615868 13173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:39.616128 13173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:39.617321 13173 hybrid_clock.cc:648] HybridClock initialized: now 1790263419617305 us; error 26 us; skew 500 ppm
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263419, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:39.620201 13173 init.cc:377] Logged in from keytab as oryx/127.4.78.65@KRBTEST.COM (short username oryx)
I20260924 15:23:39.620652 13173 webserver.cc:460] Webserver started at http://127.4.78.65:37429/ using document root <none> and password file <none>
I20260924 15:23:39.620894 13173 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:39.620975 13173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:39.622335 13173 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:39.622951 13207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.623122 13173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:39.623219 13173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "8839c6a7e00943c9933c565b4dbd6089"
format_stamp: "Formatted at 2026-09-24 15:23:38 on dist-test-slave-l766"
server_key: "51fc4291ce17cb3e61a29e64302c5143"
server_key_iv: "07f069ff2ecffcbcc3583106ecd45ced"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.623517 13173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:39.640698 13173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:39.642923 13173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:39.643078 13173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:39.643347 13173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:39.643656 13173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:39.643709 13173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.643750 13173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:39.643776 13173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.650158 13173 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:38145
I20260924 15:23:39.650202 13320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:38145 every 8 connection(s)
I20260924 15:23:39.650552 13173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:39.656271 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 13173
I20260924 15:23:39.656567 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:38639
--local_ip_for_outbound_sockets=127.4.78.66
--tserver_master_addrs=127.4.78.126:39369
--webserver_port=39433
--webserver_interface=127.4.78.66
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.78.66
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:39.660152 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.651329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:43881)
0924 15:23:39.651521 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.651574 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.652201 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.652632 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.652637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.653336 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.653608 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.653614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.654775 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.654776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.655972 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.655975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.656932 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.656934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.657076 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.657636 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.657646 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.659938 (+ 2292us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
W20260924 15:23:39.660414 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.651540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:43881 (local address 127.4.78.126:39369)
0924 15:23:39.651702 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.651710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.651746 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:23:39.652386 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.652391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.652433 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.652538 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.653730 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.654674 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.656094 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.656749 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.656841 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.660055 (+ 3214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:43881: BlockingRecv error: recv got EOF from 127.4.78.65:43881 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
W20260924 15:23:39.660440 13321 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39369 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.78.126:39369: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:39.664934 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.660665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:60821)
0924 15:23:39.660702 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.660714 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.660761 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.660911 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.660912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.661038 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.661079 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.661081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.661717 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.661719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.662298 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.662299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.662672 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.662673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.662737 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.663149 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.663154 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.664855 (+ 1701us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 15:23:39.664984 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.660713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:60821 (local address 127.4.78.126:39369)
0924 15:23:39.660745 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.660746 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.660755 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:39.660824 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.660825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.660835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.660909 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.661161 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.661636 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.662386 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.662580 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.662669 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.664924 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:60821: BlockingRecv error: recv got EOF from 127.4.78.65:60821 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:39.669426 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.665477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:58667)
0924 15:23:39.665591 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.665601 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.665669 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.665792 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.665792 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.665939 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.665964 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.665965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.666557 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.666558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.667036 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.667037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.667404 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.667405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.667457 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.667857 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.667862 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.669354 (+ 1492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260924 15:23:39.669476 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.665537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:58667 (local address 127.4.78.126:39369)
0924 15:23:39.665589 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.665590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.665655 (+ 65us) server_negotiation.cc:407] Connection header received
0924 15:23:39.665726 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.665727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.665737 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.665779 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.666037 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.666478 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.667115 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.667320 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.667402 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.669416 (+ 2014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:58667: BlockingRecv error: recv got EOF from 127.4.78.65:58667 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 15:23:39.669751 13321 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:23:39.743827 13325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:39.744016 13325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:39.744040 13325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:39.745129 13325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:39.745162 13325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:39.745186 13325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:39.746523 13325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.66.keytab
--principal=oryx/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:38639
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=39433
--tserver_master_addrs=127.4.78.126:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:39.746697 13325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:39.746855 13325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:39.751639 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:39.751690 13330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.751703 13325 server_base.cc:1034] running on GCE node
W20260924 15:23:39.751669 13331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.752028 13325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:39.752240 13325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:39.753382 13325 hybrid_clock.cc:648] HybridClock initialized: now 1790263419753362 us; error 36 us; skew 500 ppm
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263419, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:39.755739 13325 init.cc:377] Logged in from keytab as oryx/127.4.78.66@KRBTEST.COM (short username oryx)
I20260924 15:23:39.756091 13325 webserver.cc:460] Webserver started at http://127.4.78.66:39433/ using document root <none> and password file <none>
I20260924 15:23:39.756266 13325 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:39.756326 13325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:39.757514 13325 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:39.758071 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.758230 13325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:39.758315 13325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "f95535d70bbd4e4c95c14e3c161ba27f"
format_stamp: "Formatted at 2026-09-24 15:23:39 on dist-test-slave-l766"
server_key: "788999037fe98a50eb7555e032ffeb28"
server_key_iv: "935e4229f85069a58d756de780cc6164"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.758595 13325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:39.779464 13325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:39.781155 13325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:39.781309 13325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:39.781563 13325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:39.781935 13325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:39.781994 13325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.782037 13325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:39.782063 13325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.787815 13325 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:38639
I20260924 15:23:39.787858 13453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:38639 every 8 connection(s)
I20260924 15:23:39.788173 13325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:39.792713 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 13325
I20260924 15:23:39.793011 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:39081
--local_ip_for_outbound_sockets=127.4.78.67
--tserver_master_addrs=127.4.78.126:39369
--webserver_port=46031
--webserver_interface=127.4.78.67
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.78.67
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:23:39.796770 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.788843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:36875 (local address 127.4.78.126:39369)
0924 15:23:39.788908 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.788910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.789138 (+ 228us) server_negotiation.cc:407] Connection header received
0924 15:23:39.789589 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.789590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.789603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.789654 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.790670 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.791525 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.793174 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.793497 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.793559 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.796716 (+ 3157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:36875: BlockingRecv error: recv got EOF from 127.4.78.66:36875 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:23:39.796820 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.788831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:36875)
0924 15:23:39.789011 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.789056 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.789511 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.789731 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.789735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.790354 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.790552 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.790557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.791640 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.791641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.793067 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.793070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.793647 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.793648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.793801 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.794287 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.794297 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.796638 (+ 2341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":107,"threads_started":1}
W20260924 15:23:39.797035 13454 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39369 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.78.126:39369: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:39.801555 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.797248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:35903)
0924 15:23:39.797294 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.797305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.797341 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.797464 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.797465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.797594 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.797632 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.797633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.798321 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.798322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.798823 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.798825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.799195 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.799196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.799269 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.799695 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.799699 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.801479 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:23:39.801606 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.797303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:35903 (local address 127.4.78.126:39369)
0924 15:23:39.797347 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.797349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.797360 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:23:39.797392 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.797393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.797402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.797463 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.797711 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.798235 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.798918 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.799118 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.799190 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.801542 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:35903: BlockingRecv error: recv got EOF from 127.4.78.66:35903 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:39.806231 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.802061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:48505)
0924 15:23:39.802107 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.802117 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.802177 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.802331 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.802333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.802455 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.802479 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.802480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.803074 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.803075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.803629 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.803631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.803959 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.803960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.804011 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.804436 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.804441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.806167 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:23:39.806317 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.802133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:48505 (local address 127.4.78.126:39369)
0924 15:23:39.802187 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.802189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.802198 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:39.802251 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.802252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.802261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.802329 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.802552 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.802997 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.803711 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.803891 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.803962 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.806242 (+ 2280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:48505: BlockingRecv error: recv got EOF from 127.4.78.66:48505 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260924 15:23:39.806346 13454 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:23:39.876976 13457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:39.877171 13457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:39.877193 13457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:39.878315 13457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:39.878350 13457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:39.878376 13457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:39.879707 13457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/krb5kdc/oryx_127.4.78.67.keytab
--principal=oryx/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:39081
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=46031
--tserver_master_addrs=127.4.78.126:39369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:39.879884 13457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:39.880060 13457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:39.884781 13463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.884891 13457 server_base.cc:1034] running on GCE node
W20260924 15:23:39.884796 13462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:39.884792 13465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:39.885265 13457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:39.885509 13457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:39.886663 13457 hybrid_clock.cc:648] HybridClock initialized: now 1790263419886650 us; error 41 us; skew 500 ppm
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263419, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:39.888970 13457 init.cc:377] Logged in from keytab as oryx/127.4.78.67@KRBTEST.COM (short username oryx)
I20260924 15:23:39.889412 13457 webserver.cc:460] Webserver started at http://127.4.78.67:46031/ using document root <none> and password file <none>
I20260924 15:23:39.889642 13457 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:39.889729 13457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:39.891052 13457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:39.891636 13472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.891809 13457 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:39.891909 13457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "91a1faf599ef476285e336a395f360ee"
format_stamp: "Formatted at 2026-09-24 15:23:39 on dist-test-slave-l766"
server_key: "99df3f95776243a233a83659dc8d7a90"
server_key_iv: "bb0a1ab8304af4db4059ce8c9a1ad52d"
server_key_version: "encryptionkey@0"
I20260924 15:23:39.892243 13457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:39.919065 13457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:39.921037 13457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:39.921211 13457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:39.921463 13457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:39.921798 13457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:39.921885 13457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.921931 13457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:39.921958 13457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:39.928013 13457 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:39081
I20260924 15:23:39.928056 13586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:39081 every 8 connection(s)
I20260924 15:23:39.928429 13457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:39.935341 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.929086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:57851)
0924 15:23:39.929268 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.929310 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.929786 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.930058 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.930061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.930608 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.930801 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.930806 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.931475 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.931475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.932284 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.932286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.932606 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.932606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.932709 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.933130 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.933138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.935206 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":105,"threads_started":1}
W20260924 15:23:39.935339 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.929106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:57851 (local address 127.4.78.126:39369)
0924 15:23:39.929175 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.929176 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.929395 (+ 219us) server_negotiation.cc:407] Connection header received
0924 15:23:39.929934 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.929935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.929944 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.929994 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.930877 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.931381 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.932362 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.932532 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.932579 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.935291 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:57851: BlockingRecv error: recv got EOF from 127.4.78.67:57851 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 15:23:39.935616 13587 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39369 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.78.126:39369: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
I20260924 15:23:39.938432 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 13457
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:39.940151 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.935823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:33997)
0924 15:23:39.935874 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.935885 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.935929 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.936221 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.936222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.936324 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.936363 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.936364 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.937084 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.937085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.937595 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.937597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.937996 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.937997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.938045 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.938449 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.938453 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.940075 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:23:39.940213 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.936035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:33997 (local address 127.4.78.126:39369)
0924 15:23:39.936108 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.936110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.936119 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:39.936140 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.936140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.936155 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.936199 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.936425 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.937018 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.937666 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.937933 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.937984 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.940144 (+ 2160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:33997: BlockingRecv error: recv got EOF from 127.4.78.67:33997 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:39 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:39.945508 13590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.940686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:55765)
0924 15:23:39.940792 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:39.940803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:39.940852 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:39.940975 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:39.940975 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:39.941105 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:39.941132 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.941133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.941965 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.941966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.942755 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:39.942758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.943195 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.943196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:39.943234 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.943708 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:39.943732 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:39.945434 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92}
W20260924 15:23:39.945556 13324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:39.940749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:55765 (local address 127.4.78.126:39369)
0924 15:23:39.940811 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.940812 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.940839 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:23:39.940903 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.940904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.940911 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.940953 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.941195 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.941876 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.942839 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.943116 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.943189 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.945497 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:55765: BlockingRecv error: recv got EOF from 127.4.78.67:55765 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 15:23:39.945688 13587 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:23:39.946111 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:39.938833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.4.78.126:39369)
0924 15:23:39.939094 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:39.939095 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:39.939101 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:23:39.939115 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:39.939116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:39.939124 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:39.939199 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:39.939824 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.940294 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.941335 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:39.941630 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:39.941696 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:39.943001 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:39.943006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:39.943006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:39.943018 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:39.944567 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:39.944938 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:39.944941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:39.944942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:39.945068 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:39.945300 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:39.945301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:39.945301 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:39.945528 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:39.945697 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:39.945971 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 15:23:39.946057 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":56,"threads_started":1}
Sep 24 15:23:40 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:40 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:40.674647 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:40.670151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:54849)
0924 15:23:40.670330 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:40.670341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:40.670400 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:40.670526 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:40.670527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:40.670691 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:40.670739 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:40.670741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.671390 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.671391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:40.671982 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:40.671985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.672318 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.672319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:40.672360 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:40.672835 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:40.672839 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:40.674585 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":103,"threads_started":1}
W20260924 15:23:40.674747 13594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:40.670238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:54849 (local address 127.4.78.126:39369)
0924 15:23:40.670346 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:40.670349 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:40.670404 (+ 55us) server_negotiation.cc:407] Connection header received
0924 15:23:40.670441 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:40.670441 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:40.670452 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:40.670507 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:40.670804 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.671304 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.672065 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.672254 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.672304 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:40.674671 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:54849: BlockingRecv error: recv got EOF from 127.4.78.65:54849 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
Sep 24 15:23:40 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:40 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:40.811411 13595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:40.806690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:38279)
0924 15:23:40.806880 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:40.806894 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:40.806981 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:40.807137 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:40.807138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:40.807319 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:40.807368 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:40.807370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.808087 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.808088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:40.808653 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:40.808655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.809060 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.809061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:40.809127 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:40.809548 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:40.809552 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:40.811308 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":111,"threads_started":1}
W20260924 15:23:40.811445 13594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:40.806811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:38279 (local address 127.4.78.126:39369)
0924 15:23:40.806885 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:40.806888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:40.806963 (+ 75us) server_negotiation.cc:407] Connection header received
0924 15:23:40.807067 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:40.807068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:40.807081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:40.807122 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:40.807463 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.807981 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.808745 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.808963 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.809048 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:40.811382 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:38279: BlockingRecv error: recv got EOF from 127.4.78.66:38279 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 15:23:40 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:40 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:40.950630 13596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:40.946050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:43499)
0924 15:23:40.946222 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:40.946234 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:40.946328 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:40.946467 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:40.946468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:40.946648 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:40.946695 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:40.946696 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.947395 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.947396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:40.947945 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:40.947947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.948326 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.948327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:40.948391 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:40.948824 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:40.948828 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:40.950566 (+ 1738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
W20260924 15:23:40.950703 13594 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:40.946126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:43499 (local address 127.4.78.126:39369)
0924 15:23:40.946209 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:40.946212 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:40.946315 (+ 103us) server_negotiation.cc:407] Connection header received
0924 15:23:40.946386 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:40.946387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:40.946400 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:40.946455 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:40.946786 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.947302 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.948034 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:40.948234 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:40.948324 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:40.950655 (+ 2331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:43499: BlockingRecv error: recv got EOF from 127.4.78.67:43499 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 15:23:41 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:41 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:41.679551 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:41.675160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:60277)
0924 15:23:41.675321 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:41.675340 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:41.675387 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:41.675526 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:41.675526 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:41.675686 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:41.675736 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:41.675737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.676369 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.676370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:41.676874 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:41.676875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.677189 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.677190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:41.677229 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:41.677660 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:41.677664 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:41.679448 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":111,"threads_started":1}
W20260924 15:23:41.679723 13598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:41.675248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:60277 (local address 127.4.78.126:39369)
0924 15:23:41.675383 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:41.675385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:41.675421 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:23:41.675443 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:41.675443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:41.675454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:41.675503 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:41.675803 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.676303 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.676942 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.677123 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.677171 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:41.679530 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:60277: BlockingRecv error: recv got EOF from 127.4.78.65:60277 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
Sep 24 15:23:41 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:41 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:41.816581 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:41.811960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:32995)
0924 15:23:41.812135 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:41.812147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:41.812237 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:41.812370 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:41.812371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:41.812553 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:41.812599 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:41.812601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.813310 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.813311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:41.813883 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:41.813885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.814261 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.814262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:41.814320 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:41.814757 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:41.814762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:41.816467 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
W20260924 15:23:41.816635 13598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:41.812044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:32995 (local address 127.4.78.126:39369)
0924 15:23:41.812118 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:41.812120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:41.812219 (+ 99us) server_negotiation.cc:407] Connection header received
0924 15:23:41.812297 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:41.812298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:41.812311 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:41.812357 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:41.812701 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.813203 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.813971 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.814172 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.814259 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:41.816567 (+ 2308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:32995: BlockingRecv error: recv got EOF from 127.4.78.66:32995 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 15:23:41 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:41 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:41.955513 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:41.951083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:49639)
0924 15:23:41.951211 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:41.951230 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:41.951306 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:41.951471 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:41.951471 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:41.951642 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:41.951690 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:41.951692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.952353 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.952354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:41.952873 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:41.952875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.953245 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.953246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:41.953308 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:41.953700 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:41.953704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:41.955408 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
W20260924 15:23:41.955549 13598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:41.951226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:49639 (local address 127.4.78.126:39369)
0924 15:23:41.951300 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:41.951302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:41.951311 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:41.951369 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:41.951370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:41.951398 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:41.951464 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:41.951772 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.952263 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.952961 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:41.953160 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:41.953243 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:41.955485 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:49639: BlockingRecv error: recv got EOF from 127.4.78.67:49639 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 15:23:42 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:42 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:42.684975 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:42.680108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:59979)
0924 15:23:42.680230 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:42.680242 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:42.680303 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:42.680486 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:42.680487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:42.680644 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:42.680694 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:42.680696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.681519 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.681520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:42.682262 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:42.682266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.682655 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.682656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:42.682718 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:42.683134 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:42.683139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:42.684907 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":71,"threads_started":1}
W20260924 15:23:42.685120 13602 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:42.680199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:59979 (local address 127.4.78.126:39369)
0924 15:23:42.680349 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:42.680350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:42.680360 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:42.680391 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:42.680392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:42.680403 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:42.680482 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:42.680783 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.681443 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.682349 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.682551 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.682650 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:42.685005 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:59979: BlockingRecv error: recv got EOF from 127.4.78.65:59979 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
Sep 24 15:23:42 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:42 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:42.821478 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:42.817044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:45077)
0924 15:23:42.817216 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:42.817226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:42.817278 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:42.817400 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:42.817401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:42.817548 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:42.817604 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:42.817605 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.818259 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.818260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:42.818790 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:42.818791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.819134 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.819135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:42.819173 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:42.819627 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:42.819631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:42.821404 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
W20260924 15:23:42.821552 13602 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:42.817147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:45077 (local address 127.4.78.126:39369)
0924 15:23:42.817223 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:42.817224 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:42.817266 (+ 42us) server_negotiation.cc:407] Connection header received
0924 15:23:42.817323 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:42.817324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:42.817334 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:42.817384 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:42.817665 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.818195 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.818867 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.819056 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.819117 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:42.821504 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:45077: BlockingRecv error: recv got EOF from 127.4.78.66:45077 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 15:23:42 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:42 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:42.960886 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:42.955978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:51581)
0924 15:23:42.956155 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:42.956166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:42.956241 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:42.956407 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:42.956408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:42.956589 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:42.956646 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:42.956648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.957341 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.957342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:42.957994 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:42.957997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.958366 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.958367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:42.958425 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:42.958867 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:42.958871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:42.960781 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
W20260924 15:23:42.960923 13602 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:42.956064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:51581 (local address 127.4.78.126:39369)
0924 15:23:42.956135 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:42.956137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:42.956232 (+ 95us) server_negotiation.cc:407] Connection header received
0924 15:23:42.956306 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:42.956308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:42.956328 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:42.956386 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:42.956724 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.957243 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.958075 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:42.958266 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:42.958352 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:42.960856 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:51581: BlockingRecv error: recv got EOF from 127.4.78.67:51581 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 15:23:43 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:43 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:43.693539 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:43.685392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:43241)
0924 15:23:43.685580 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:43.685592 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:43.685656 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:43.685855 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:43.685856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:43.686029 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:43.686082 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:43.686083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.686724 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.686725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:43.687251 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:43.687253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.687589 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.687590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:43.687629 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:43.691673 (+ 4044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:43.691678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:43.693479 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":88,"threads_started":1}
W20260924 15:23:43.693694 13606 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:43.685524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:43241 (local address 127.4.78.126:39369)
0924 15:23:43.685652 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:43.685660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:43.685669 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:43.685715 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:43.685716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:43.685727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:43.685788 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:43.686153 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.686659 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.687326 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.687523 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.687576 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:43.693596 (+ 6020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:43241: BlockingRecv error: recv got EOF from 127.4.78.65:43241 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":65,"threads_started":1}
Sep 24 15:23:43 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:43 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:43.826858 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:43.822120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:35697)
0924 15:23:43.822255 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:43.822266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:43.822320 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:43.822465 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:43.822466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:43.822628 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:43.822678 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:43.822680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.823315 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.823316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:43.823836 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:43.823838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.824149 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.824150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:43.824192 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:43.824700 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:43.824705 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:43.826801 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
W20260924 15:23:43.826993 13606 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:43.822192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:35697 (local address 127.4.78.126:39369)
0924 15:23:43.822291 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:43.822294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:43.822306 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:23:43.822380 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:43.822381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:43.822393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:43.822434 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:43.822742 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.823249 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.823907 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.824087 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.824135 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:43.826888 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:35697: BlockingRecv error: recv got EOF from 127.4.78.66:35697 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 24 15:23:43 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:43 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:43.966107 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:43.961416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:47457)
0924 15:23:43.961555 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:43.961573 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:43.961640 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:43.961771 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:43.961772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:43.961961 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:43.962009 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:43.962011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.962621 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.962622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:43.963134 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:43.963136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.963451 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.963451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:43.963490 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:43.963928 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:43.963933 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:43.966039 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
W20260924 15:23:43.966198 13606 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:43.961485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:47457 (local address 127.4.78.126:39369)
0924 15:23:43.961549 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:43.961552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:43.961675 (+ 123us) server_negotiation.cc:407] Connection header received
0924 15:23:43.961699 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:43.961699 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:43.961711 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:43.961752 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:43.962074 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.962554 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.963209 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:43.963386 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:43.963439 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:43.966115 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:47457: BlockingRecv error: recv got EOF from 127.4.78.67:47457 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 15:23:44 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:44 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:44.698709 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:44.694095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:53843)
0924 15:23:44.694253 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:44.694264 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:44.694317 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:44.694477 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:44.694478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:44.694640 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:44.694707 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:44.694709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.695495 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.695496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:44.696030 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:44.696031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.696389 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.696390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:44.696437 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:44.696922 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:44.696927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:44.698640 (+ 1713us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":111,"threads_started":1}
W20260924 15:23:44.698885 13610 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:44.694187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:53843 (local address 127.4.78.126:39369)
0924 15:23:44.694339 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:44.694342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:44.694350 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:44.694380 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:44.694381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:44.694391 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:44.694468 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:44.694773 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.695423 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.696104 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.696324 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.696376 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:44.698781 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:53843: BlockingRecv error: recv got EOF from 127.4.78.65:53843 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
Sep 24 15:23:44 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:44 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:44.832423 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:44.827563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:55903)
0924 15:23:44.827712 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:44.827740 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:44.827803 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:44.827973 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:44.827974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:44.828150 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:44.828203 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:44.828205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.828931 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.828932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:44.829636 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:44.829639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.830126 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.830127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:44.830187 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:44.830605 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:44.830610 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:44.832367 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
W20260924 15:23:44.832531 13610 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:44.827703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:55903 (local address 127.4.78.126:39369)
0924 15:23:44.827772 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:44.827774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:44.827786 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:23:44.827882 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:44.827883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:44.827895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:44.827968 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:44.828281 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.828840 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.829737 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.830037 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.830123 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:44.832457 (+ 2334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:55903: BlockingRecv error: recv got EOF from 127.4.78.66:55903 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 15:23:44 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:44 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:44.971258 13612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:44.966576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:46751)
0924 15:23:44.966716 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:44.966726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:44.966792 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:44.966982 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:44.966983 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:44.967143 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:44.967195 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:44.967196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.967869 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.967870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:44.968395 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:44.968397 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.968801 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.968802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:44.968872 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:44.969418 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:44.969423 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:44.971200 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
W20260924 15:23:44.971352 13610 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:44.966654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:46751 (local address 127.4.78.126:39369)
0924 15:23:44.966745 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:44.966747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:44.966770 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:23:44.966852 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:44.966853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:44.966867 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:44.966916 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:44.967257 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.967798 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.968466 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:44.968720 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:44.968799 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:44.971285 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:46751: BlockingRecv error: recv got EOF from 127.4.78.67:46751 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 24 15:23:45 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:45 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:45.704001 13613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:45.699218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:44923)
0924 15:23:45.699335 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:45.699348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:45.699424 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:45.699552 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:45.699553 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:45.699712 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:45.699761 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:45.699763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.700398 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.700399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:45.701084 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:45.701088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.701508 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.701509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:45.701554 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:45.702014 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:45.702019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:45.703889 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
W20260924 15:23:45.704041 13614 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:45.699296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:44923 (local address 127.4.78.126:39369)
0924 15:23:45.699425 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:45.699426 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:45.699435 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:45.699472 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:45.699473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:45.699483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:45.699533 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:45.699833 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.700331 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.701170 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.701440 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.701494 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:45.703967 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:44923: BlockingRecv error: recv got EOF from 127.4.78.65:44923 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
Sep 24 15:23:45 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:45 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:45.837399 13615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:45.832893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:41153)
0924 15:23:45.833023 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:45.833034 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:45.833112 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:45.833259 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:45.833260 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:45.833418 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:45.833464 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:45.833466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.834156 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.834157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:45.834653 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:45.834654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.834966 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.834967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:45.835005 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:45.835453 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:45.835458 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:45.837341 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
W20260924 15:23:45.837757 13614 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:45.832939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:41153 (local address 127.4.78.126:39369)
0924 15:23:45.833011 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:45.833014 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:45.833115 (+ 101us) server_negotiation.cc:407] Connection header received
0924 15:23:45.833173 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:45.833173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:45.833186 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:45.833228 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:45.833560 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.834053 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.834721 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.834901 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.834955 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:45.837689 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:41153: BlockingRecv error: recv got EOF from 127.4.78.66:41153 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 15:23:45 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:45 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:45.976742 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:45.971754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:41381)
0924 15:23:45.972089 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:45.972098 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:45.972185 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:45.972340 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:45.972341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:45.972521 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:45.972571 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:45.972573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.973248 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.973250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:45.973784 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:45.973785 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.974195 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.974197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:45.974265 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:45.974810 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:45.974815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:45.976648 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":85,"threads_started":1}
W20260924 15:23:45.976799 13614 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:45.971858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:41381 (local address 127.4.78.126:39369)
0924 15:23:45.972036 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:45.972039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:45.972165 (+ 126us) server_negotiation.cc:407] Connection header received
0924 15:23:45.972241 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:45.972242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:45.972258 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:45.972333 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:45.972657 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.973157 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.973913 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:45.974109 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:45.974194 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:45.976733 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:41381: BlockingRecv error: recv got EOF from 127.4.78.67:41381 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
Sep 24 15:23:46 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:46 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:46.708842 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:46.704458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:50809)
0924 15:23:46.704586 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:46.704598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:46.704661 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:46.704828 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:46.704829 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:46.704986 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:46.705034 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:46.705036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.705675 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.705676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:46.706218 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:46.706220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.706542 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.706542 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:46.706581 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:46.707020 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:46.707024 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:46.708785 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":58,"threads_started":1}
W20260924 15:23:46.708992 13618 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:46.704535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:50809 (local address 127.4.78.126:39369)
0924 15:23:46.704673 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:46.704675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:46.704696 (+ 21us) server_negotiation.cc:407] Connection header received
0924 15:23:46.704720 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:46.704721 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:46.704731 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:46.704821 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:46.705101 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.705606 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.706288 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.706479 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.706530 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:46.708900 (+ 2370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:50809: BlockingRecv error: recv got EOF from 127.4.78.65:50809 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
Sep 24 15:23:46 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:46 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:46.842698 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:46.838059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:36375)
0924 15:23:46.838210 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:46.838222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:46.838320 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:46.838476 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:46.838477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:46.838656 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:46.838711 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:46.838712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.839395 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.839396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:46.839950 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:46.839952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.840323 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.840324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:46.840382 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:46.840842 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:46.840846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:46.842577 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":72,"threads_started":1}
W20260924 15:23:46.842751 13618 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:46.838115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:36375 (local address 127.4.78.126:39369)
0924 15:23:46.838203 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:46.838205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:46.838301 (+ 96us) server_negotiation.cc:407] Connection header received
0924 15:23:46.838378 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:46.838379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:46.838392 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:46.838468 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:46.838794 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.839308 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.840041 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.840249 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.840327 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:46.842672 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:36375: BlockingRecv error: recv got EOF from 127.4.78.66:36375 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:23:46 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:46 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:46.981731 13620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:46.977237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:43271)
0924 15:23:46.977384 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:46.977394 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:46.977452 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:46.977586 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:46.977586 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:46.977743 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:46.977792 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:46.977794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.978489 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.978490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:46.979032 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:46.979033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.979340 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.979341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:46.979386 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:46.979796 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:46.979800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:46.981671 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260924 15:23:46.981881 13618 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:46.977302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:43271 (local address 127.4.78.126:39369)
0924 15:23:46.977363 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:46.977365 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:46.977457 (+ 92us) server_negotiation.cc:407] Connection header received
0924 15:23:46.977497 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:46.977497 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:46.977511 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:46.977570 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:46.977916 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.978405 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.979103 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:46.979271 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:46.979318 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:46.981792 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:43271: BlockingRecv error: recv got EOF from 127.4.78.67:43271 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 15:23:47 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:47 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:47.713555 13621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:47.709283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:37799)
0924 15:23:47.709409 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:47.709424 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:47.709471 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:47.709670 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:47.709671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:47.709852 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:47.709902 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:47.709904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.710544 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.710545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:47.711058 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:47.711059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.711373 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.711373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:47.711414 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:47.711842 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:47.711846 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:47.713497 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
W20260924 15:23:47.713699 13622 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:47.709398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:37799 (local address 127.4.78.126:39369)
0924 15:23:47.709546 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:47.709547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:47.709555 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:47.709576 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:47.709577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:47.709586 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:47.709643 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:47.709975 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.710475 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.711126 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.711309 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.711357 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:47.713610 (+ 2253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:37799: BlockingRecv error: recv got EOF from 127.4.78.65:37799 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
Sep 24 15:23:47 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:47 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:47.848006 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:47.843198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:36689)
0924 15:23:47.843358 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:47.843369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:47.843452 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:47.843590 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:47.843591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:47.843769 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:47.843820 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:47.843821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.844673 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.844674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:47.845188 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:47.845189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.845544 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.845545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:47.845612 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:47.846142 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:47.846149 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:47.847910 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
W20260924 15:23:47.848054 13622 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:47.843257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:36689 (local address 127.4.78.126:39369)
0924 15:23:47.843333 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:47.843335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:47.843433 (+ 98us) server_negotiation.cc:407] Connection header received
0924 15:23:47.843520 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:47.843521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:47.843533 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:47.843584 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:47.843895 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.844559 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.845271 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.845470 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.845545 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:47.847989 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:36689: BlockingRecv error: recv got EOF from 127.4.78.66:36689 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 15:23:47 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:47 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:47.987282 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:47.982478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:41299)
0924 15:23:47.982651 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:47.982661 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:47.982734 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:47.982899 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:47.982900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:47.983106 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:47.983172 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:47.983174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.984038 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.984039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:47.984635 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:47.984636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.984960 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.984960 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:47.984999 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:47.985435 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:47.985439 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:47.987226 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
W20260924 15:23:47.987412 13622 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:47.982544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:41299 (local address 127.4.78.126:39369)
0924 15:23:47.982597 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:47.982599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:47.982767 (+ 168us) server_negotiation.cc:407] Connection header received
0924 15:23:47.982811 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:47.982811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:47.982826 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:47.982882 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:47.983282 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.983970 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.984711 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:47.984893 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:47.984949 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:47.987324 (+ 2375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:41299: BlockingRecv error: recv got EOF from 127.4.78.67:41299 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 24 15:23:48 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:48 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:48.723435 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:48.714010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:40645)
0924 15:23:48.714190 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:48.714204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:48.714270 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:48.714456 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:48.714457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:48.714618 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:48.714665 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:48.714667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.715460 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.715462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:48.716166 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:48.716169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.716498 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.716498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:48.716536 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:48.721479 (+ 4943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:48.721484 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:48.723368 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":76,"threads_started":1}
W20260924 15:23:48.723562 13626 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:48.714221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:40645 (local address 127.4.78.126:39369)
0924 15:23:48.714337 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:48.714338 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:48.714346 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:48.714367 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:48.714368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:48.714377 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:48.714435 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:48.714728 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.715347 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.716240 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.716434 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.716485 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:48.723455 (+ 6970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:40645: BlockingRecv error: recv got EOF from 127.4.78.65:40645 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":59,"threads_started":1}
Sep 24 15:23:48 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:48 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:48.853096 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:48.848630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:58929)
0924 15:23:48.848818 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:48.848828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:48.848890 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:48.849024 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:48.849024 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:48.849217 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:48.849265 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:48.849266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.849904 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.849905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:48.850437 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:48.850439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.850769 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.850770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:48.850810 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:48.851329 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:48.851334 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:48.853040 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":75,"threads_started":1}
W20260924 15:23:48.853173 13626 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:48.848701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:58929 (local address 127.4.78.126:39369)
0924 15:23:48.848823 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:48.848825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:48.848890 (+ 65us) server_negotiation.cc:407] Connection header received
0924 15:23:48.848938 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:48.848939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:48.848950 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:48.849000 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:48.849329 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.849843 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.850505 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.850700 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.850756 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:48.853128 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:58929: BlockingRecv error: recv got EOF from 127.4.78.66:58929 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 24 15:23:48 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:48 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:48.992221 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:48.987787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:51723)
0924 15:23:48.987912 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:48.987922 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:48.987988 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:48.988122 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:48.988123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:48.988331 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:48.988382 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:48.988384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.989003 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.989004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:48.989534 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:48.989536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.989891 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.989892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:48.989947 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:48.990375 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:48.990379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:48.992154 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
W20260924 15:23:48.992321 13626 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:48.987843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:51723 (local address 127.4.78.126:39369)
0924 15:23:48.987911 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:48.987913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:48.987990 (+ 77us) server_negotiation.cc:407] Connection header received
0924 15:23:48.988033 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:48.988034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:48.988047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:48.988100 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:48.988453 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.988939 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.989603 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:48.989780 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:48.989846 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:48.992253 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:51723: BlockingRecv error: recv got EOF from 127.4.78.67:51723 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 15:23:49 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:49 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:49.729117 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:49.724063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:60769)
0924 15:23:49.724222 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:49.724233 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:49.724282 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:49.724433 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:49.724434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:49.724593 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:49.724641 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:49.724643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.725321 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.725323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:49.725982 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:49.726018 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.726400 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.726400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:49.726459 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:49.727027 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:49.727033 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:49.729008 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
W20260924 15:23:49.729182 13630 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:49.724156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:60769 (local address 127.4.78.126:39369)
0924 15:23:49.724308 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:49.724310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:49.724318 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:23:49.724341 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:49.724342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:49.724352 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:49.724418 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:49.724712 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.725210 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.726117 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.726331 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.726385 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:49.729106 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:60769: BlockingRecv error: recv got EOF from 127.4.78.65:60769 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
Sep 24 15:23:49 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:49 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:49.858465 13631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:49.853674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:44243)
0924 15:23:49.853872 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:49.853883 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:49.853970 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:49.854141 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:49.854142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:49.854324 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:49.854374 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:49.854376 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.855043 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.855045 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:49.855570 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:49.855571 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.855945 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.855946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:49.856015 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:49.856472 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:49.856476 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:49.858369 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":100,"threads_started":1}
W20260924 15:23:49.858517 13630 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:49.853762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:44243 (local address 127.4.78.126:39369)
0924 15:23:49.853855 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:49.853857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:49.853950 (+ 93us) server_negotiation.cc:407] Connection header received
0924 15:23:49.854039 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:49.854040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:49.854052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:49.854118 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:49.854456 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.854958 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.855657 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.855858 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.855942 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:49.858452 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:44243: BlockingRecv error: recv got EOF from 127.4.78.66:44243 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 24 15:23:49 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:49 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:49.997525 13632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:49.992788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:56117)
0924 15:23:49.992919 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:49.992938 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:49.993013 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:49.993160 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:49.993161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:49.993350 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:49.993398 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:49.993400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.994110 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.994111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:49.994647 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:49.994649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.995043 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.995044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:49.995105 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:49.995576 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:49.995580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:49.997420 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
W20260924 15:23:49.997745 13630 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:49.992938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:56117 (local address 127.4.78.126:39369)
0924 15:23:49.993006 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:49.993007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:49.993017 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:49.993073 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:49.993074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:49.993092 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:49.993155 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:49.993484 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.994029 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.994740 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:49.994970 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:49.995045 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:49.997524 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:56117: BlockingRecv error: recv got EOF from 127.4.78.67:56117 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:23:50 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:50 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:50.734349 13633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:50.729595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:57493)
0924 15:23:50.729747 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:50.729763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:50.729832 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:50.729979 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:50.729980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:50.730143 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:50.730196 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:50.730198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.730830 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.730831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:50.731373 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:50.731375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.731695 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.731695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:50.731734 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:50.732239 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:50.732244 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:50.734240 (+ 1996us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
W20260924 15:23:50.734472 13634 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:50.729670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:57493 (local address 127.4.78.126:39369)
0924 15:23:50.729803 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:50.729840 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:50.729852 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:23:50.729885 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:50.729886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:50.729897 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:50.729959 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:50.730262 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.730756 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.731445 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.731630 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.731678 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:50.734374 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:57493: BlockingRecv error: recv got EOF from 127.4.78.65:57493 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
Sep 24 15:23:50 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:50 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:50.866429 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:50.859096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:46313)
0924 15:23:50.861176 (+ 2080us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:50.861200 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:50.861278 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:50.861463 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:50.861465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:50.861691 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:50.861758 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:50.861761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.862528 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.862529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:50.863500 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:50.863503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.863849 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.863850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:50.863919 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:50.864373 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:50.864379 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:50.866361 (+ 1982us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2056,"thread_start_us":92,"threads_started":1}
W20260924 15:23:50.866657 13634 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:50.859272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:46313 (local address 127.4.78.126:39369)
0924 15:23:50.859344 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:50.859347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:50.861267 (+ 1920us) server_negotiation.cc:407] Connection header received
0924 15:23:50.861367 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:50.861369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:50.861384 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:50.861441 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:50.861905 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.862436 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.863585 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.863776 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.863826 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:50.866482 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:46313: BlockingRecv error: recv got EOF from 127.4.78.66:46313 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 15:23:50 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:51 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:51.002458 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:50.998041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:51871)
0924 15:23:50.998206 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:50.998217 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:50.998272 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:50.998415 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:50.998416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:50.998587 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:50.998653 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:50.998655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.999271 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.999272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:50.999798 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:50.999800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.000101 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.000102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:51.000149 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:51.000567 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:51.000572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:51.002391 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":102,"threads_started":1}
W20260924 15:23:51.002578 13634 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:50.998110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:51871 (local address 127.4.78.126:39369)
0924 15:23:50.998186 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:50.998188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:50.998274 (+ 86us) server_negotiation.cc:407] Connection header received
0924 15:23:50.998326 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:50.998327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:50.998339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:50.998390 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:50.998722 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:50.999205 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:50.999866 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.000041 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.000089 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:51.002481 (+ 2392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:51871: BlockingRecv error: recv got EOF from 127.4.78.67:51871 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 15:23:51 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:51 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:51.740093 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:51.735024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:54729)
0924 15:23:51.735216 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:51.735226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:51.735276 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:51.735432 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:51.735433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:51.735597 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:51.735646 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:51.735648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.736351 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.736352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:51.737162 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:51.737165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.737503 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.737503 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:51.737544 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:51.738188 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:51.738198 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:51.739996 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
W20260924 15:23:51.740147 13638 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:51.735086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:54729 (local address 127.4.78.126:39369)
0924 15:23:51.735262 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:51.735269 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:51.735311 (+ 42us) server_negotiation.cc:407] Connection header received
0924 15:23:51.735332 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:51.735333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:51.735342 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:51.735413 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:51.735709 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.736252 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.737228 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.737432 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.737486 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:51.740075 (+ 2589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:54729: BlockingRecv error: recv got EOF from 127.4.78.65:54729 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
Sep 24 15:23:51 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:51 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:51.871376 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:51.866937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:51075)
0924 15:23:51.867052 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:51.867065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:51.867127 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:51.867237 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:51.867238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:51.867400 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:51.867448 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:51.867449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.868082 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.868082 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:51.868609 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:51.868611 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.868932 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.868932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:51.868975 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:51.869430 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:51.869435 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:51.871257 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
W20260924 15:23:51.871418 13638 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:51.866969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:51075 (local address 127.4.78.126:39369)
0924 15:23:51.867050 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:51.867052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:51.867128 (+ 76us) server_negotiation.cc:407] Connection header received
0924 15:23:51.867173 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:51.867174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:51.867184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:51.867232 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:51.867505 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.868009 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.868681 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:51.868868 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:51.868917 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:51.871344 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:51075: BlockingRecv error: recv got EOF from 127.4.78.66:51075 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 15:23:52 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:52 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:52.007880 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:52.003007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:33523)
0924 15:23:52.003130 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:52.003141 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:52.003204 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:52.003351 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:52.003370 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:52.003553 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:52.003603 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:52.003604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.004382 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.004383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:52.004949 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:52.004951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.005288 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.005288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:52.005328 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:52.005765 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:52.005770 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:52.007770 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
W20260924 15:23:52.007920 13638 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:52.003111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:33523 (local address 127.4.78.126:39369)
0924 15:23:52.003176 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:52.003180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:52.003195 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:23:52.003251 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:52.003252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:52.003264 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:52.003321 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:52.003676 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.004310 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.005026 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.005218 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.005285 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:52.007851 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:33523: BlockingRecv error: recv got EOF from 127.4.78.67:33523 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 15:23:52 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:52 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:52.745365 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:52.740547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:48257)
0924 15:23:52.740747 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:52.740757 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:52.740835 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:52.741108 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:52.741110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:52.741369 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:52.741431 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:52.741433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.742159 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.742160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:52.742698 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:52.742699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.743042 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.743043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:52.743081 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:52.743505 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:52.743510 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:52.745298 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":56,"threads_started":1}
W20260924 15:23:52.745455 13642 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:52.740765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:48257 (local address 127.4.78.126:39369)
0924 15:23:52.740881 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:52.740885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:52.740914 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:23:52.740940 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:52.740941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:52.740956 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:52.741055 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:52.741512 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.742094 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.742765 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.742979 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.743028 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:52.745381 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:48257: BlockingRecv error: recv got EOF from 127.4.78.65:48257 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
Sep 24 15:23:52 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:52 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:52.876225 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:52.871852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:37643)
0924 15:23:52.871996 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:52.872009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:52.872073 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:52.872210 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:52.872211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:52.872381 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:52.872429 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:52.872431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.873045 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.873046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:52.873584 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:52.873585 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.873953 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.873954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:52.873996 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:52.874399 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:52.874404 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:52.876156 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":70,"threads_started":1}
W20260924 15:23:52.876345 13642 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:52.871971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:37643 (local address 127.4.78.126:39369)
0924 15:23:52.872028 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:52.872030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:52.872059 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:23:52.872131 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:52.872132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:52.872142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:52.872192 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:52.872490 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.872983 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.873652 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:52.873887 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:52.873945 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:52.876249 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:37643: BlockingRecv error: recv got EOF from 127.4.78.66:37643 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 15:23:53 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:53 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:53.012964 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:53.008432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:35953)
0924 15:23:53.008604 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:53.008615 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:53.008673 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:53.008802 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:53.008802 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:53.008965 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:53.009013 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:53.009014 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.009662 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.009663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:53.010208 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:53.010210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.010525 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.010526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:53.010563 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:53.011162 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:53.011169 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:53.012897 (+ 1728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":109,"threads_started":1}
W20260924 15:23:53.013049 13642 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:53.008476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:35953 (local address 127.4.78.126:39369)
0924 15:23:53.008542 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:53.008544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:53.008674 (+ 130us) server_negotiation.cc:407] Connection header received
0924 15:23:53.008713 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:53.008714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:53.008727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:53.008774 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:53.009105 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.009599 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.010276 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.010457 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.010510 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:53.012984 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:35953: BlockingRecv error: recv got EOF from 127.4.78.67:35953 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 15:23:53 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:53 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:53.754212 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:53.745928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:46537)
0924 15:23:53.746072 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:53.746085 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:53.746144 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:53.746286 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:53.746287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:53.746445 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:53.746499 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:53.746501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.747133 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.747133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:53.747645 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:53.747646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.747957 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.747958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:53.747998 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:53.752196 (+ 4198us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:53.752202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:53.754154 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
W20260924 15:23:53.754302 13646 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:53.745961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:46537 (local address 127.4.78.126:39369)
0924 15:23:53.746066 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:53.746068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:53.746130 (+ 62us) server_negotiation.cc:407] Connection header received
0924 15:23:53.746196 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:53.746196 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:53.746207 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:53.746255 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:53.746561 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.747069 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.747711 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.747897 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.747944 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:53.754214 (+ 6270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:46537: BlockingRecv error: recv got EOF from 127.4.78.65:46537 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
Sep 24 15:23:53 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:53 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:53.881105 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:53.876633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:34969)
0924 15:23:53.876791 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:53.876802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:53.876863 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:53.876980 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:53.876980 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:53.877144 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:53.877201 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:53.877203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.877842 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.877843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:53.878372 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:53.878374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.878693 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.878694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:53.878734 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:53.879259 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:53.879264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:53.880995 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
W20260924 15:23:53.881145 13646 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:53.876727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:34969 (local address 127.4.78.126:39369)
0924 15:23:53.876790 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:53.876792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:53.876851 (+ 59us) server_negotiation.cc:407] Connection header received
0924 15:23:53.876911 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:53.876911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:53.876922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:53.876975 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:53.877260 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.877752 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.878440 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:53.878626 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:53.878678 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:53.881079 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:34969: BlockingRecv error: recv got EOF from 127.4.78.66:34969 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 15:23:54 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:54 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.67@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:54.018255 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:54.013630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.67:51813)
0924 15:23:54.013775 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:54.013787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:54.013853 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:54.014044 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:54.014045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:54.014239 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:54.014288 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:54.014289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.014950 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.014950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:54.015493 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:54.015494 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.015800 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.015800 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:54.015840 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:54.016332 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:54.016337 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:54.018185 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
W20260924 15:23:54.018384 13646 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:54.013696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:51813 (local address 127.4.78.126:39369)
0924 15:23:54.013829 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:54.013831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:54.013841 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:54.013942 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:54.013942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:54.013962 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:54.014022 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:54.014355 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.014868 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.015563 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.015738 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.015789 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:54.018301 (+ 2512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.67:51813: BlockingRecv error: recv got EOF from 127.4.78.67:51813 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 15:23:54 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:54 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.65@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:54.759558 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:54.754801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.65:59097)
0924 15:23:54.754957 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:54.754971 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:54.755066 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:54.755227 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:54.755228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:54.755385 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:54.755437 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:54.755439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.756099 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.756100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:54.756636 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:54.756638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.757063 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.757065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:54.757183 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:54.757799 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:54.757828 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:54.759501 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
W20260924 15:23:54.759665 13650 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:54.754872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:59097 (local address 127.4.78.126:39369)
0924 15:23:54.755022 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:54.755024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:54.755033 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:23:54.755135 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:54.755136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:54.755147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:54.755207 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:54.755502 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.756003 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.756717 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.756937 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.757008 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:54.759590 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.65:59097: BlockingRecv error: recv got EOF from 127.4.78.65:59097 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
Sep 24 15:23:54 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:23:54 dist-test-slave-l766 krb5kdc[12588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.78.66@KRBTEST.COM for oryx/127.4.78.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:23:54.885874 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:54.881667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39369 (local address 127.4.78.66:55163)
0924 15:23:54.881830 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:54.881849 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:54.881904 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:54.882044 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:54.882045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:54.882203 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:54.882253 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:54.882255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.882876 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.882877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:54.883393 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:54.883394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.883701 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.883702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:54.883740 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:54.884165 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:54.884169 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:54.885790 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.78.126:39369: Server oryx/127.4.78.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
W20260924 15:23:54.886023 13650 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:54.881752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:55163 (local address 127.4.78.126:39369)
0924 15:23:54.881857 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:54.881859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:54.881892 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:23:54.881967 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:54.881968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:54.881978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:54.882022 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:54.882327 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.882811 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.883462 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:54.883639 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:54.883686 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:54.885948 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.78.66:55163: BlockingRecv error: recv got EOF from 127.4.78.66:55163 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:23:54.939725 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 13173
I20260924 15:23:54.944449 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 13325
I20260924 15:23:54.948559 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 13457
I20260924 15:23:54.953203 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 13103
2026-09-24T15:23:54Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17401 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T15:23:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:54Z Disabled control of system clock
I20260924 15:23:54.980201 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42713
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 15:23:55.059491 13655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:55.059710 13655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:55.059742 13655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:55.060907 13655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:55.060969 13655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:55.060990 13655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:55.061020 13655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:55.061044 13655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:55.062450 13655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:42713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:55.062716 13655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:55.062990 13655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:55.067592 13663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.067595 13661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.067868 13655 server_base.cc:1034] running on GCE node
W20260924 15:23:55.067592 13660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.068197 13655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:55.068506 13655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:55.069648 13655 hybrid_clock.cc:648] HybridClock initialized: now 1790263435069635 us; error 27 us; skew 500 ppm
I20260924 15:23:55.070854 13655 webserver.cc:460] Webserver started at http://127.4.78.126:40363/ using document root <none> and password file <none>
I20260924 15:23:55.071079 13655 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:55.071122 13655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:55.071256 13655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:55.072312 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "2e13612c9b1d4c3d8817a151583cf0bc"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "7bf7a25e1d0a9671bdc268f568755c28"
server_key_iv: "7673e213050f600774ef60fc9c83fb9b"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.072659 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "2e13612c9b1d4c3d8817a151583cf0bc"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "7bf7a25e1d0a9671bdc268f568755c28"
server_key_iv: "7673e213050f600774ef60fc9c83fb9b"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.073954 13655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:55.074653 13669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.074843 13655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:55.074934 13655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "2e13612c9b1d4c3d8817a151583cf0bc"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "7bf7a25e1d0a9671bdc268f568755c28"
server_key_iv: "7673e213050f600774ef60fc9c83fb9b"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.075017 13655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:55.086781 13655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:55.087044 13655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:55.087183 13655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:55.090996 13655 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:42713
I20260924 15:23:55.091034 13721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:42713 every 8 connection(s)
I20260924 15:23:55.091363 13655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:55.091984 13722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:55.094226 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 13655
I20260924 15:23:55.094292 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:55.094463 4409 external_mini_cluster.cc:1428] Setting key 51dd88743720bc5b97e842df425f7602
I20260924 15:23:55.094556 13722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc: Bootstrap starting.
I20260924 15:23:55.095263 13722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:55.095579 13722 log.cc:824] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:55.096283 13722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc: No bootstrap required, opened a new log
I20260924 15:23:55.098481 13722 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e13612c9b1d4c3d8817a151583cf0bc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42713 } }
I20260924 15:23:55.098661 13722 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:55.098698 13722 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e13612c9b1d4c3d8817a151583cf0bc, State: Initialized, Role: FOLLOWER
I20260924 15:23:55.098836 13722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e13612c9b1d4c3d8817a151583cf0bc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42713 } }
I20260924 15:23:55.098949 13722 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:55.098973 13722 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:55.099041 13722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:55.099676 13722 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e13612c9b1d4c3d8817a151583cf0bc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42713 } }
I20260924 15:23:55.099800 13722 leader_election.cc:302] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e13612c9b1d4c3d8817a151583cf0bc; no voters:
I20260924 15:23:55.099987 13722 leader_election.cc:288] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:55.100018 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.094971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.4.78.126:42713)
0924 15:23:55.095149 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:55.095156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:55.095180 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:23:55.095619 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:55.095625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:55.095719 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:55.095867 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:55.096082 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.096964 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.098126 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.098504 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.098596 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:55.099290 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:55.099294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:55.099294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:55.099461 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:55.099491 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:55.099500 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:55.099548 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:55.099654 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 15:23:55.099719 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 15:23:55.100041 13727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:55.100246 13722 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:55.100272 13727 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [term 1 LEADER]: Becoming Leader. State: Replica: 2e13612c9b1d4c3d8817a151583cf0bc, State: Running, Role: LEADER
I20260924 15:23:55.100346 13727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e13612c9b1d4c3d8817a151583cf0bc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42713 } }
I20260924 15:23:55.100755 13728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e13612c9b1d4c3d8817a151583cf0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e13612c9b1d4c3d8817a151583cf0bc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42713 } } }
I20260924 15:23:55.100853 13729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e13612c9b1d4c3d8817a151583cf0bc. Latest consensus state: current_term: 1 leader_uuid: "2e13612c9b1d4c3d8817a151583cf0bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e13612c9b1d4c3d8817a151583cf0bc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42713 } } }
I20260924 15:23:55.100888 13728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:55.100934 13729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:55.101320 13736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:55.101811 13736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:55.103385 13736 catalog_manager.cc:1347] Generated new cluster ID: 2b30c7f1399f41a9a400db505e5e5996
I20260924 15:23:55.103440 13736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:55.120085 13736 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:55.120757 13736 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:55.125284 13736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e13612c9b1d4c3d8817a151583cf0bc: Generated new TSK 0
I20260924 15:23:55.125439 13736 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:23:55.130030 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42713
--builtin_ntp_servers=127.4.78.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 15:23:55.215085 13746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:55.215389 13746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:55.215428 13746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:55.217228 13746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:55.217288 13746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:55.217325 13746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:55.219173 13746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:55.219420 13746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:55.219735 13746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:55.225394 13752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.225392 13751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.225502 13746 server_base.cc:1034] running on GCE node
W20260924 15:23:55.225455 13754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.225980 13746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:55.226269 13746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:55.227440 13746 hybrid_clock.cc:648] HybridClock initialized: now 1790263435227421 us; error 31 us; skew 500 ppm
I20260924 15:23:55.228972 13746 webserver.cc:460] Webserver started at http://127.4.78.65:42435/ using document root <none> and password file <none>
I20260924 15:23:55.229250 13746 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:55.229318 13746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:55.229454 13746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:55.230811 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "a106f3b204f34c229f9530aac10044cb"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "3fadfe6c38636c67cd3fa3d2314dde88"
server_key_iv: "836fa3ea4d4258a432f0cc991b7d68a0"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.231184 13746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "a106f3b204f34c229f9530aac10044cb"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "3fadfe6c38636c67cd3fa3d2314dde88"
server_key_iv: "836fa3ea4d4258a432f0cc991b7d68a0"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.232693 13746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:55.233517 13760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.233711 13746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.233794 13746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "a106f3b204f34c229f9530aac10044cb"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "3fadfe6c38636c67cd3fa3d2314dde88"
server_key_iv: "836fa3ea4d4258a432f0cc991b7d68a0"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.233882 13746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:55.245249 13746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:55.245462 13746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:55.245553 13746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:55.245697 13746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:55.246037 13746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:55.246070 13746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.246090 13746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:55.246102 13746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.251607 13746 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:44755
I20260924 15:23:55.251682 13873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:44755 every 8 connection(s)
I20260924 15:23:55.251998 13746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:55.254981 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 13746
I20260924 15:23:55.255050 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:55.255232 4409 external_mini_cluster.cc:1428] Setting key 1587d4461249464de71589f81b67f4a2
I20260924 15:23:55.256191 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42713
--builtin_ntp_servers=127.4.78.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 15:23:55.257560 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.252759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:38587 (local address 127.4.78.126:42713)
0924 15:23:55.252821 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:55.252824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:55.253076 (+ 252us) server_negotiation.cc:407] Connection header received
0924 15:23:55.253539 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:55.253540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:55.253548 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:55.253604 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:55.254285 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.254759 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.255997 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.256204 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.256289 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:55.257268 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:55.257270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:55.257270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:55.257276 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:55.257294 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:55.257300 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:55.257365 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:55.257482 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 15:23:55.257505 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:23:55.257844 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.252739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42713 (local address 127.4.78.65:38587)
0924 15:23:55.252951 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:55.252990 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:55.253461 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:55.253655 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:55.253658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:55.253665 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:55.254213 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:55.254218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.254857 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.254858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:55.255900 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:55.255903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.256368 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.256369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:55.256524 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:55.257101 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:55.257102 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:55.257129 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:55.257151 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:55.257381 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:55.257401 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:55.257404 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:55.257477 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:23:55.257588 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260924 15:23:55.258344 13874 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42713
I20260924 15:23:55.258440 13874 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:55.258635 13874 heartbeater.cc:511] Master 127.4.78.126:42713 requested a full tablet report, sending...
I20260924 15:23:55.259124 13686 ts_manager.cc:194] Registered new tserver with Master: a106f3b204f34c229f9530aac10044cb (127.4.78.65:44755)
I20260924 15:23:55.259822 13686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.65:38587
W20260924 15:23:55.336274 13877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:55.336457 13877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:55.336478 13877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:55.337827 13877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:55.337877 13877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:55.337910 13877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:55.339273 13877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:55.339447 13877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:55.339691 13877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:55.344432 13883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.344496 13882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.344523 13885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.344648 13877 server_base.cc:1034] running on GCE node
I20260924 15:23:55.344908 13877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:55.345165 13877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:55.346346 13877 hybrid_clock.cc:648] HybridClock initialized: now 1790263435346333 us; error 29 us; skew 500 ppm
I20260924 15:23:55.347541 13877 webserver.cc:460] Webserver started at http://127.4.78.66:34807/ using document root <none> and password file <none>
I20260924 15:23:55.347795 13877 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:55.347875 13877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:55.348021 13877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:55.349012 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "8594b08175f34a1c82084265ee5edcb9"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "091a11611b5f129adff2566c35562e8f"
server_key_iv: "6c8b1bac3f9b61553d3f267fb9fa1abd"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.349406 13877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "8594b08175f34a1c82084265ee5edcb9"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "091a11611b5f129adff2566c35562e8f"
server_key_iv: "6c8b1bac3f9b61553d3f267fb9fa1abd"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.350746 13877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:55.351508 13891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.351679 13877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:55.351769 13877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "8594b08175f34a1c82084265ee5edcb9"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "091a11611b5f129adff2566c35562e8f"
server_key_iv: "6c8b1bac3f9b61553d3f267fb9fa1abd"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.351856 13877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:55.361954 13877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:55.362224 13877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:55.362367 13877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:55.362576 13877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:55.362892 13877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:55.362946 13877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.362986 13877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:55.363015 13877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.368788 13877 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:42137
I20260924 15:23:55.368824 14004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:42137 every 8 connection(s)
I20260924 15:23:55.369141 13877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:55.371364 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 13877
I20260924 15:23:55.371464 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:55.371629 4409 external_mini_cluster.cc:1428] Setting key 23303b4b317538b0f5d87c461f7c04a5
I20260924 15:23:55.372546 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42713
--builtin_ntp_servers=127.4.78.84:45781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 15:23:55.374493 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.369880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:45485 (local address 127.4.78.126:42713)
0924 15:23:55.369933 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:55.369935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:55.370157 (+ 222us) server_negotiation.cc:407] Connection header received
0924 15:23:55.370611 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:55.370612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:55.370622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:55.370678 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:55.371323 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.371982 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.373047 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.373257 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.373348 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:55.374249 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:55.374251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:55.374251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:55.374255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:55.374270 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:55.374274 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:55.374319 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:55.374415 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:23:55.374436 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:23:55.374758 14007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.369852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42713 (local address 127.4.78.66:45485)
0924 15:23:55.370032 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:55.370069 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:55.370534 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:55.370728 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:55.370731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:55.370739 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:55.371217 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:55.371221 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.372080 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.372081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:55.372961 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:55.372964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.373359 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.373360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:55.373488 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:55.374089 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:55.374089 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:55.374130 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:55.374159 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:55.374338 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:55.374342 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:55.374345 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:55.374413 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 15:23:55.374523 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 15:23:55.375301 14005 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42713
I20260924 15:23:55.375381 14005 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:55.375572 14005 heartbeater.cc:511] Master 127.4.78.126:42713 requested a full tablet report, sending...
I20260924 15:23:55.375973 13686 ts_manager.cc:194] Registered new tserver with Master: 8594b08175f34a1c82084265ee5edcb9 (127.4.78.66:42137)
I20260924 15:23:55.376353 13686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.66:45485
W20260924 15:23:55.453229 14008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:55.453452 14008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:55.453486 14008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:55.454720 14008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:55.454785 14008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:55.454829 14008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:55.456146 14008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:55.456331 14008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:55.456555 14008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:55.461297 14013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.461295 14016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.461346 14014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.461761 14008 server_base.cc:1034] running on GCE node
I20260924 15:23:55.461994 14008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:55.462172 14008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:55.463305 14008 hybrid_clock.cc:648] HybridClock initialized: now 1790263435463295 us; error 25 us; skew 500 ppm
I20260924 15:23:55.464450 14008 webserver.cc:460] Webserver started at http://127.4.78.67:40287/ using document root <none> and password file <none>
I20260924 15:23:55.464636 14008 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:55.464676 14008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:55.464771 14008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:55.465677 14008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "e6335d1f05634353b272c89cfeb33ad6"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "255aad11d973ba7d7d6d5b6060cc80bc"
server_key_iv: "4b432c0bc44923e629770944ecefb90d"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.466017 14008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "e6335d1f05634353b272c89cfeb33ad6"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "255aad11d973ba7d7d6d5b6060cc80bc"
server_key_iv: "4b432c0bc44923e629770944ecefb90d"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.467176 14008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:23:55.467839 14022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.468037 14008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:23:55.468137 14008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "e6335d1f05634353b272c89cfeb33ad6"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "255aad11d973ba7d7d6d5b6060cc80bc"
server_key_iv: "4b432c0bc44923e629770944ecefb90d"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.468210 14008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:55.480804 14008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:55.481078 14008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:55.481220 14008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:55.481431 14008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:55.481742 14008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:55.481782 14008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.481853 14008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:55.481899 14008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.487401 14008 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:37477
I20260924 15:23:55.487491 14135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:37477 every 8 connection(s)
I20260924 15:23:55.487772 14008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:55.487955 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 14008
I20260924 15:23:55.488050 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:55.488283 4409 external_mini_cluster.cc:1428] Setting key 0f70873bf35990575747714a4ae6aa96
I20260924 15:23:55.493151 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.488489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:33875 (local address 127.4.78.126:42713)
0924 15:23:55.488532 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:55.488534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:55.488930 (+ 396us) server_negotiation.cc:407] Connection header received
0924 15:23:55.489462 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:55.489463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:55.489475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:55.489525 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:55.490255 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.490779 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.491768 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.491979 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.492054 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:55.492898 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:55.492900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:55.492900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:55.492904 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:55.492919 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:55.492923 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:55.492969 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:55.493053 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:23:55.493077 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:23:55.493360 14138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.488511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42713 (local address 127.4.78.67:33875)
0924 15:23:55.488730 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:55.488771 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:55.489344 (+ 573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:55.489588 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:55.489592 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:55.489602 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:55.490155 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:55.490160 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.490871 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.490872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:55.491673 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:55.491675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:55.492059 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:55.492061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:55.492203 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:55.492748 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:55.492748 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:55.492775 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:55.492795 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:55.492968 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:55.492973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:55.492976 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:55.493045 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:23:55.493140 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260924 15:23:55.493794 14136 heartbeater.cc:347] Connected to a master server at 127.4.78.126:42713
I20260924 15:23:55.493919 14136 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:55.494087 14136 heartbeater.cc:511] Master 127.4.78.126:42713 requested a full tablet report, sending...
I20260924 15:23:55.494432 13686 ts_manager.cc:194] Registered new tserver with Master: e6335d1f05634353b272c89cfeb33ad6 (127.4.78.67:37477)
I20260924 15:23:55.494822 13686 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.67:33875
I20260924 15:23:55.500247 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:23:55.502163 13725 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:55.501601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.4.78.126:42713)
0924 15:23:55.501645 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:55.501646 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:55.501781 (+ 135us) server_negotiation.cc:407] Connection header received
0924 15:23:55.501885 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:55.501886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:55.501896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:55.501964 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:55.502099 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41456: BlockingRecv error: recv got EOF from 127.0.0.1:41456 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:23:55.502264 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 13746
I20260924 15:23:55.508797 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 13877
I20260924 15:23:55.513031 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 14008
I20260924 15:23:55.517127 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 13655
2026-09-24T15:23:55Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (562 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T15:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:55Z Disabled control of system clock
I20260924 15:23:55.543047 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:38721
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:38721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:23:55.622341 14147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:55.622541 14147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:55.622561 14147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:55.623600 14147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:55.623631 14147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:55.623643 14147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:55.623656 14147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:55.623667 14147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:55.624903 14147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:38721
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:38721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:55.625087 14147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:55.625326 14147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:55.630115 14155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.630110 14152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.630200 14147 server_base.cc:1034] running on GCE node
W20260924 15:23:55.630110 14153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.630604 14147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:55.630890 14147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:55.632056 14147 hybrid_clock.cc:648] HybridClock initialized: now 1790263435632042 us; error 25 us; skew 500 ppm
I20260924 15:23:55.633217 14147 webserver.cc:460] Webserver started at http://127.4.78.126:41945/ using document root <none> and password file <none>
I20260924 15:23:55.633453 14147 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:55.633522 14147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:55.633695 14147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:55.634804 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "03817b5c65594af5a22fd56eccffd5dd"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "9cffc5fee499abb62912b54ff358ac61"
server_key_iv: "ad77b3a3c53adab42fa1da43cdae49c3"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.635169 14147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "03817b5c65594af5a22fd56eccffd5dd"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "9cffc5fee499abb62912b54ff358ac61"
server_key_iv: "ad77b3a3c53adab42fa1da43cdae49c3"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.636420 14147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:55.637151 14161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.637305 14147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:55.637392 14147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "03817b5c65594af5a22fd56eccffd5dd"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "9cffc5fee499abb62912b54ff358ac61"
server_key_iv: "ad77b3a3c53adab42fa1da43cdae49c3"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.637454 14147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:55.640834 14147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:55.641053 14147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:55.641147 14147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:55.645022 14147 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:38721
I20260924 15:23:55.645052 14213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:38721 every 8 connection(s)
I20260924 15:23:55.645391 14147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:55.646127 14214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:55.646780 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 14147
I20260924 15:23:55.646885 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:55.647090 4409 external_mini_cluster.cc:1428] Setting key b6d5efd4ceb3819c03389f65d972864b
I20260924 15:23:55.649632 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.647680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.4.78.126:38721)
0924 15:23:55.647852 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:55.647859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:55.647881 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:23:55.648325 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:55.648332 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:55.648373 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:55.648487 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:55.649180 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:55.649183 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:55.649184 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:55.649187 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:55.649226 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:55.649235 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:55.649297 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:55.649383 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 15:23:55.649461 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
I20260924 15:23:55.650151 14214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd: Bootstrap starting.
I20260924 15:23:55.650746 14214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:55.651018 14214 log.cc:824] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:55.651662 14214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd: No bootstrap required, opened a new log
I20260924 15:23:55.652943 14214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03817b5c65594af5a22fd56eccffd5dd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38721 } }
I20260924 15:23:55.653095 14214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:55.653142 14214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03817b5c65594af5a22fd56eccffd5dd, State: Initialized, Role: FOLLOWER
I20260924 15:23:55.653270 14214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03817b5c65594af5a22fd56eccffd5dd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38721 } }
I20260924 15:23:55.653371 14214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:55.653416 14214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:55.653463 14214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:55.654031 14214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03817b5c65594af5a22fd56eccffd5dd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38721 } }
I20260924 15:23:55.654151 14214 leader_election.cc:302] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03817b5c65594af5a22fd56eccffd5dd; no voters:
I20260924 15:23:55.654317 14214 leader_election.cc:288] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:55.654568 14214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:55.654584 14219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:55.654716 14219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [term 1 LEADER]: Becoming Leader. State: Replica: 03817b5c65594af5a22fd56eccffd5dd, State: Running, Role: LEADER
I20260924 15:23:55.654842 14219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03817b5c65594af5a22fd56eccffd5dd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38721 } }
I20260924 15:23:55.655203 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03817b5c65594af5a22fd56eccffd5dd. Latest consensus state: current_term: 1 leader_uuid: "03817b5c65594af5a22fd56eccffd5dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03817b5c65594af5a22fd56eccffd5dd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38721 } } }
I20260924 15:23:55.655320 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:55.655642 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03817b5c65594af5a22fd56eccffd5dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03817b5c65594af5a22fd56eccffd5dd" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 38721 } } }
I20260924 15:23:55.655719 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:55.655747 14228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:55.656175 14228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:23:55.657629 14228 catalog_manager.cc:1347] Generated new cluster ID: 206378d35fce44fa8c292d2aeae7625e
I20260924 15:23:55.657684 14228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:55.671537 14228 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:55.671711 14228 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:55.679767 14228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03817b5c65594af5a22fd56eccffd5dd: Generated new TSK 0
I20260924 15:23:55.679946 14228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:23:55.684731 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38721
--builtin_ntp_servers=127.4.78.84:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:23:55.766028 14238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:55.766214 14238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:55.766237 14238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:55.767388 14238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:55.767427 14238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:55.767452 14238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:55.768761 14238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:55.769030 14238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:55.769345 14238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:55.775247 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.775280 14238 server_base.cc:1034] running on GCE node
W20260924 15:23:55.775259 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.775449 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.775676 14238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:55.775874 14238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:55.777011 14238 hybrid_clock.cc:648] HybridClock initialized: now 1790263435777000 us; error 31 us; skew 500 ppm
I20260924 15:23:55.778224 14238 webserver.cc:460] Webserver started at http://127.4.78.65:45203/ using document root <none> and password file <none>
I20260924 15:23:55.778474 14238 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:55.778544 14238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:55.778699 14238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:55.779850 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "890faf48335e40ff98eea8b9d1cc52ce"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "61aa26d81793668223070c80ed49f874"
server_key_iv: "ca24c22d94cb6f58c2e0775106f76487"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.780203 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "890faf48335e40ff98eea8b9d1cc52ce"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "61aa26d81793668223070c80ed49f874"
server_key_iv: "ca24c22d94cb6f58c2e0775106f76487"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.781478 14238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:55.782266 14252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.782439 14238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.782543 14238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "890faf48335e40ff98eea8b9d1cc52ce"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "61aa26d81793668223070c80ed49f874"
server_key_iv: "ca24c22d94cb6f58c2e0775106f76487"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.782617 14238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:55.785872 14238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:55.786132 14238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:55.786266 14238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:55.786474 14238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:55.786790 14238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:55.786846 14238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.786885 14238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:55.786913 14238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.792637 14238 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:44315
I20260924 15:23:55.792678 14365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:44315 every 8 connection(s)
I20260924 15:23:55.792977 14238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:55.796170 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.793637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:37099 (local address 127.4.78.126:38721)
0924 15:23:55.793709 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:55.793711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:55.793991 (+ 280us) server_negotiation.cc:407] Connection header received
0924 15:23:55.794572 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:55.794573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:55.794581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:55.794655 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:55.795940 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:55.795941 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:55.795941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:55.795946 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:55.795960 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:55.795963 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:55.796020 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:55.796087 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 15:23:55.796102 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:23:55.796366 14368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.793604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38721 (local address 127.4.78.65:37099)
0924 15:23:55.793858 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:55.793897 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:55.794442 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:55.794724 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:55.794728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:55.794738 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:55.795792 (+ 1054us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:55.795793 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:55.795819 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:55.795845 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:55.796020 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:55.796025 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:55.796028 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:55.796100 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:23:55.796157 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":106,"threads_started":1}
I20260924 15:23:55.796902 14366 heartbeater.cc:347] Connected to a master server at 127.4.78.126:38721
I20260924 15:23:55.796990 14366 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:55.797145 14366 heartbeater.cc:511] Master 127.4.78.126:38721 requested a full tablet report, sending...
I20260924 15:23:55.797640 14178 ts_manager.cc:194] Registered new tserver with Master: 890faf48335e40ff98eea8b9d1cc52ce (127.4.78.65:44315)
I20260924 15:23:55.799376 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 14238
I20260924 15:23:55.799445 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:55.799640 4409 external_mini_cluster.cc:1428] Setting key 4b800cf23db94ca8092d26aac763d25e
I20260924 15:23:55.800487 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38721
--builtin_ntp_servers=127.4.78.84:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:23:55.879834 14369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:55.880069 14369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:55.880107 14369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:55.881695 14369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:55.881739 14369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:55.881779 14369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:55.883108 14369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:55.883342 14369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:55.883637 14369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:55.888465 14377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.888512 14374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:55.888477 14375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:55.888676 14369 server_base.cc:1034] running on GCE node
I20260924 15:23:55.888904 14369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:55.889143 14369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:55.890306 14369 hybrid_clock.cc:648] HybridClock initialized: now 1790263435890300 us; error 35 us; skew 500 ppm
I20260924 15:23:55.891503 14369 webserver.cc:460] Webserver started at http://127.4.78.66:40835/ using document root <none> and password file <none>
I20260924 15:23:55.891746 14369 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:55.891813 14369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:55.891955 14369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:55.892958 14369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "c8e85e83b5ee4e298f44bec6b7015d5e"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "98adb062567ed74368ea73cfe6ab6364"
server_key_iv: "3cae213fa49a10c7b70157c63fbff4c3"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.893318 14369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "c8e85e83b5ee4e298f44bec6b7015d5e"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "98adb062567ed74368ea73cfe6ab6364"
server_key_iv: "3cae213fa49a10c7b70157c63fbff4c3"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.894681 14369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:23:55.895421 14383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.895596 14369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:55.895692 14369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "c8e85e83b5ee4e298f44bec6b7015d5e"
format_stamp: "Formatted at 2026-09-24 15:23:55 on dist-test-slave-l766"
server_key: "98adb062567ed74368ea73cfe6ab6364"
server_key_iv: "3cae213fa49a10c7b70157c63fbff4c3"
server_key_version: "encryptionkey@0"
I20260924 15:23:55.895768 14369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:55.899034 14369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:55.899293 14369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:55.899436 14369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:55.899648 14369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:55.899937 14369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:55.899991 14369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.900027 14369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:55.900053 14369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:55.905782 14369 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:46349
I20260924 15:23:55.905869 14497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:46349 every 8 connection(s)
I20260924 15:23:55.906198 14369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:55.908946 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.906946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:58155 (local address 127.4.78.126:38721)
0924 15:23:55.906988 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:55.906990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:55.907181 (+ 191us) server_negotiation.cc:407] Connection header received
0924 15:23:55.907630 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:55.907630 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:55.907636 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:55.907695 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:55.908765 (+ 1070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:55.908766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:55.908767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:55.908771 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:55.908787 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:55.908791 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:55.908837 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:55.908881 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 15:23:55.908892 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:23:55.909152 14500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:55.906892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38721 (local address 127.4.78.66:58155)
0924 15:23:55.907070 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:55.907109 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:55.907544 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:55.907751 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:55.907755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:55.907762 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:55.908605 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:55.908606 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:55.908632 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:55.908660 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:55.908834 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:55.908837 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:55.908839 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:23:55.908888 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:23:55.908933 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260924 15:23:55.909631 14498 heartbeater.cc:347] Connected to a master server at 127.4.78.126:38721
I20260924 15:23:55.909721 14498 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:55.909878 14498 heartbeater.cc:511] Master 127.4.78.126:38721 requested a full tablet report, sending...
I20260924 15:23:55.910226 14178 ts_manager.cc:194] Registered new tserver with Master: c8e85e83b5ee4e298f44bec6b7015d5e (127.4.78.66:46349)
I20260924 15:23:55.915062 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 14369
I20260924 15:23:55.915153 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:55.915357 4409 external_mini_cluster.cc:1428] Setting key b2879a487c54fd6942c059e5cc81494e
I20260924 15:23:55.916157 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38721
--builtin_ntp_servers=127.4.78.84:43381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:23:55.994450 14501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:55.994676 14501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:55.994724 14501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:55.995877 14501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:55.995939 14501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:55.995982 14501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:55.997365 14501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:38721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:55.997633 14501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:55.997980 14501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:56.003408 14506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:56.003540 14501 server_base.cc:1034] running on GCE node
W20260924 15:23:56.003417 14509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:56.003417 14507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:56.003856 14501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:56.004086 14501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:56.005259 14501 hybrid_clock.cc:648] HybridClock initialized: now 1790263436005245 us; error 30 us; skew 500 ppm
I20260924 15:23:56.006474 14501 webserver.cc:460] Webserver started at http://127.4.78.67:43839/ using document root <none> and password file <none>
I20260924 15:23:56.006727 14501 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:56.006803 14501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:56.006958 14501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:56.007989 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "7f98632f751948af8838a45cc1a2827d"
format_stamp: "Formatted at 2026-09-24 15:23:56 on dist-test-slave-l766"
server_key: "af9534ca9bf622d572839c723471bdef"
server_key_iv: "6eef217606c80a5d0146c0f3e802aa74"
server_key_version: "encryptionkey@0"
I20260924 15:23:56.008342 14501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "7f98632f751948af8838a45cc1a2827d"
format_stamp: "Formatted at 2026-09-24 15:23:56 on dist-test-slave-l766"
server_key: "af9534ca9bf622d572839c723471bdef"
server_key_iv: "6eef217606c80a5d0146c0f3e802aa74"
server_key_version: "encryptionkey@0"
I20260924 15:23:56.009579 14501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:23:56.010363 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:56.010521 14501 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:56.010607 14501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "7f98632f751948af8838a45cc1a2827d"
format_stamp: "Formatted at 2026-09-24 15:23:56 on dist-test-slave-l766"
server_key: "af9534ca9bf622d572839c723471bdef"
server_key_iv: "6eef217606c80a5d0146c0f3e802aa74"
server_key_version: "encryptionkey@0"
I20260924 15:23:56.010656 14501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:56.013862 14501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:56.014060 14501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:56.014145 14501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:56.014287 14501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:56.014505 14501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:56.014532 14501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:56.014550 14501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:56.014562 14501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:56.019958 14501 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:37471
I20260924 15:23:56.020018 14629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:37471 every 8 connection(s)
I20260924 15:23:56.020315 14501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:56.020569 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 14501
I20260924 15:23:56.020663 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:56.020859 4409 external_mini_cluster.cc:1428] Setting key 85bf1ee0b1dc08ff58a9b6581e5b97c5
I20260924 15:23:56.023242 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:56.021041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:43405 (local address 127.4.78.126:38721)
0924 15:23:56.021094 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:56.021097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:56.021297 (+ 200us) server_negotiation.cc:407] Connection header received
0924 15:23:56.021803 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:56.021805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:56.021841 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:56.021895 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:56.022981 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:56.022982 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:56.022982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:23:56.022986 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:56.023002 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:23:56.023006 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:56.023076 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:56.023156 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:23:56.023171 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:23:56.023448 14632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:56.020971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:38721 (local address 127.4.78.67:43405)
0924 15:23:56.021178 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:56.021222 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:56.021682 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:56.021969 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:56.021973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:56.021982 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:56.022825 (+ 843us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:23:56.022825 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:56.022852 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:23:56.022878 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:56.023076 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:56.023081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:56.023084 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:56.023167 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:23:56.023233 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260924 15:23:56.023900 14630 heartbeater.cc:347] Connected to a master server at 127.4.78.126:38721
I20260924 15:23:56.024004 14630 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:56.024168 14630 heartbeater.cc:511] Master 127.4.78.126:38721 requested a full tablet report, sending...
I20260924 15:23:56.024482 14178 ts_manager.cc:194] Registered new tserver with Master: 7f98632f751948af8838a45cc1a2827d (127.4.78.67:37471)
I20260924 15:23:56.032635 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:23:56.034451 14217 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:23:56.034044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40810 (local address 127.4.78.126:38721)
0924 15:23:56.034092 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:56.034094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:56.034194 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:23:56.034239 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:56.034240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:56.034244 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:56.034291 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:56.034409 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40810: BlockingRecv error: recv got EOF from 127.0.0.1:40810 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:23:56.034578 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 14238
I20260924 15:23:56.039384 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 14369
I20260924 15:23:56.043799 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 14501
I20260924 15:23:56.047859 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 14147
2026-09-24T15:23:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (531 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:56 dist-test-slave-l766 krb5kdc[14640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:56 dist-test-slave-l766 krb5kdc[14640](info): set up 2 sockets
Sep 24 15:23:56 dist-test-slave-l766 krb5kdc[14640](info): commencing operation
krb5kdc: starting...
W20260924 15:23:57.073412 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:23:57.190222 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:36575
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:57.272115 14653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:57.272315 14653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:57.272337 14653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:57.273489 14653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:23:57.273531 14653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:57.273546 14653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:57.273597 14653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:23:57.273614 14653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:23:57.275158 14653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:36575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:57.275413 14653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:57.275640 14653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:57.280128 14661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:57.280176 14659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:57.280295 14653 server_base.cc:1034] running on GCE node
W20260924 15:23:57.280346 14658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:57.280606 14653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:57.280830 14653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:57.281988 14653 hybrid_clock.cc:648] HybridClock initialized: now 1790263437281982 us; error 34 us; skew 500 ppm
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:57.284348 14653 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:23:57.284821 14653 webserver.cc:460] Webserver started at http://127.4.78.126:43625/ using document root <none> and password file <none>
I20260924 15:23:57.285055 14653 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:57.285121 14653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:57.285254 14653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:57.286319 14653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "b3abc4f6d2d2ece0aa35c468e84edc10"
server_key_iv: "79c35cf69e24bd4b77c8444bdceed0fb"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.286679 14653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "b3abc4f6d2d2ece0aa35c468e84edc10"
server_key_iv: "79c35cf69e24bd4b77c8444bdceed0fb"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.287949 14653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:57.288638 14668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.288810 14653 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:57.288903 14653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "b3abc4f6d2d2ece0aa35c468e84edc10"
server_key_iv: "79c35cf69e24bd4b77c8444bdceed0fb"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.288986 14653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:57.304106 14653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:57.306416 14653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:57.306576 14653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:57.310770 14653 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:36575
I20260924 15:23:57.310804 14720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:36575 every 8 connection(s)
I20260924 15:23:57.311132 14653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:23:57.312062 14721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:57.314393 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc: Bootstrap starting.
I20260924 15:23:57.314625 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 14653
I20260924 15:23:57.314730 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:23:57.314956 4409 external_mini_cluster.cc:1428] Setting key 9981eedcf8f8c6ca801fee42c264f63a
I20260924 15:23:57.315198 14721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:57.315490 14721 log.cc:824] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:57.316180 14721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc: No bootstrap required, opened a new log
I20260924 15:23:57.318006 14721 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36575 } }
I20260924 15:23:57.318153 14721 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:57.318198 14721 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e1bee00c21400bb4ec7c4a12c1d1fc, State: Initialized, Role: FOLLOWER
I20260924 15:23:57.318341 14721 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36575 } }
I20260924 15:23:57.318431 14721 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:23:57.318473 14721 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:23:57.318518 14721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:57.319039 14721 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36575 } }
I20260924 15:23:57.319157 14721 leader_election.cc:302] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4e1bee00c21400bb4ec7c4a12c1d1fc; no voters:
I20260924 15:23:57.319358 14721 leader_election.cc:288] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:23:57.319377 14726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:57.319489 14726 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [term 1 LEADER]: Becoming Leader. State: Replica: d4e1bee00c21400bb4ec7c4a12c1d1fc, State: Running, Role: LEADER
I20260924 15:23:57.319572 14726 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36575 } }
I20260924 15:23:57.319613 14721 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:23:57.319911 14726 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36575 } } }
I20260924 15:23:57.319984 14726 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [sys.catalog]: This master's current role is: LEADER
I20260924 15:23:57.320111 14726 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4e1bee00c21400bb4ec7c4a12c1d1fc. Latest consensus state: current_term: 1 leader_uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4e1bee00c21400bb4ec7c4a12c1d1fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36575 } } }
I20260924 15:23:57.320173 14726 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc [sys.catalog]: This master's current role is: LEADER
W20260924 15:23:57.321367 14742 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:23:57.321429 14742 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:23:57.321471 14731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:23:57.321609 14731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:57.323161 14731 catalog_manager.cc:1347] Generated new cluster ID: 457f7b6c0d894ae0868c65d60aa5b17c
I20260924 15:23:57.323215 14731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:23:57.326136 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.315627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58030 (local address 127.4.78.126:36575)
0924 15:23:57.315805 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.315815 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.315850 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:23:57.316248 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.316254 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.316336 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.316496 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:57.317103 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.317996 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.318936 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.319440 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.319572 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.322545 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:57.322549 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:57.322549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:57.322560 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:57.324507 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.324873 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.324876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.324876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.324986 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.325216 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.325217 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.325218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.325531 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:57.325687 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.325765 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.325929 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260924 15:23:57.348632 14731 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:23:57.349120 14731 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:23:57.357765 14731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc: Generated new TSK 0
I20260924 15:23:57.357961 14731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:23:57.386642 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36575
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:57.471402 14747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:57.471627 14747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:57.471668 14747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:57.472774 14747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:57.472836 14747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:57.472878 14747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:23:57.474278 14747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:57.474520 14747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:57.474732 14747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:57.479475 14755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:57.479687 14747 server_base.cc:1034] running on GCE node
W20260924 15:23:57.479482 14752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:57.479509 14753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:57.479990 14747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:57.480224 14747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:57.481411 14747 hybrid_clock.cc:648] HybridClock initialized: now 1790263437481390 us; error 36 us; skew 500 ppm
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:57.483825 14747 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:23:57.484241 14747 webserver.cc:460] Webserver started at http://127.4.78.65:34203/ using document root <none> and password file <none>
I20260924 15:23:57.484469 14747 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:57.484537 14747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:57.484685 14747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:57.485747 14747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "11ab9c3fc245426e868a982ce8740e58"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "91e846d046f40dbdc780e5349713dece"
server_key_iv: "95a51c85f788bdffd7bfddbbe41df63c"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.486188 14747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "11ab9c3fc245426e868a982ce8740e58"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "91e846d046f40dbdc780e5349713dece"
server_key_iv: "95a51c85f788bdffd7bfddbbe41df63c"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.487502 14747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:23:57.488176 14762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.488358 14747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:57.488458 14747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "11ab9c3fc245426e868a982ce8740e58"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "91e846d046f40dbdc780e5349713dece"
server_key_iv: "95a51c85f788bdffd7bfddbbe41df63c"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.488535 14747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:57.507616 14747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:57.509692 14747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:57.509891 14747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:57.510135 14747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:57.510455 14747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:57.510488 14747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.510530 14747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:57.510557 14747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.516165 14747 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:42335
I20260924 15:23:57.516248 14875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:42335 every 8 connection(s)
I20260924 15:23:57.516558 14747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:23:57.521322 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 14747
I20260924 15:23:57.521427 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:23:57.521620 4409 external_mini_cluster.cc:1428] Setting key bbc26cfa6cde2797edaacf1ebd39f4e4
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:57.526365 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.517321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:52695 (local address 127.4.78.126:36575)
0924 15:23:57.517403 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.517405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.517524 (+ 119us) server_negotiation.cc:407] Connection header received
0924 15:23:57.518000 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.518001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.518011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.518064 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:57.518926 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.519421 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.520481 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.520831 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.520895 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.523531 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:57.523532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:57.523533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:57.523539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:57.524913 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.525382 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.525383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.525383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.525431 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.525698 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.525699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.525699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.525792 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:57.525904 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.526229 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.526301 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 15:23:57.526634 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.517247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:36575 (local address 127.4.78.65:52695)
0924 15:23:57.517400 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.517435 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:57.517918 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:57.518116 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:57.518119 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:57.518665 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:57.518853 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.518857 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.519497 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.519498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.520405 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.520407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.520911 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.520912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.521013 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.521583 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:57.521591 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:57.523437 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:57.524985 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:57.524992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:57.524994 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:57.525308 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:57.525490 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:57.525491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:57.525492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:57.525632 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:57.525904 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:57.525910 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:57.526082 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 15:23:57.526207 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 15:23:57.526394 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260924 15:23:57.527184 14876 heartbeater.cc:347] Connected to a master server at 127.4.78.126:36575
I20260924 15:23:57.527285 14876 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:57.527469 14876 heartbeater.cc:511] Master 127.4.78.126:36575 requested a full tablet report, sending...
I20260924 15:23:57.527959 14685 ts_manager.cc:194] Registered new tserver with Master: 11ab9c3fc245426e868a982ce8740e58 (127.4.78.65:42335)
I20260924 15:23:57.528507 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:52695
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:23:57.552364 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36575
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:57.632495 14881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:57.632726 14881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:57.632763 14881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:57.633937 14881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:57.634003 14881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:57.634045 14881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:23:57.635505 14881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:57.635763 14881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:57.635989 14881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:57.640774 14886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:57.640848 14887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:57.640954 14881 server_base.cc:1034] running on GCE node
W20260924 15:23:57.640795 14889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:57.641242 14881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:57.641530 14881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:57.642673 14881 hybrid_clock.cc:648] HybridClock initialized: now 1790263437642658 us; error 27 us; skew 500 ppm
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:57.645068 14881 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:23:57.645519 14881 webserver.cc:460] Webserver started at http://127.4.78.66:36717/ using document root <none> and password file <none>
I20260924 15:23:57.645712 14881 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:57.645753 14881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:57.645910 14881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:57.646888 14881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "658a229c789b40079910e16d5c4fc527"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "806a5e2fc3bbeaf2085a8ef0c4363fb3"
server_key_iv: "0fdfd864baa298c388947fc96a8351ba"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.647186 14881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "658a229c789b40079910e16d5c4fc527"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "806a5e2fc3bbeaf2085a8ef0c4363fb3"
server_key_iv: "0fdfd864baa298c388947fc96a8351ba"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.648391 14881 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:57.649001 14896 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.649176 14881 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:57.649250 14881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "658a229c789b40079910e16d5c4fc527"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "806a5e2fc3bbeaf2085a8ef0c4363fb3"
server_key_iv: "0fdfd864baa298c388947fc96a8351ba"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.649309 14881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:57.667086 14881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:57.668790 14881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:57.668900 14881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:57.669097 14881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:57.669355 14881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:57.669386 14881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.669407 14881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:57.669420 14881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.675122 14881 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:44865
I20260924 15:23:57.675173 15009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:44865 every 8 connection(s)
I20260924 15:23:57.675477 14881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:23:57.676600 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 14881
I20260924 15:23:57.676695 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:23:57.676895 4409 external_mini_cluster.cc:1428] Setting key aa407405e991c0d82270a4daee1c1599
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
I20260924 15:23:57.686329 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.676223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:38613 (local address 127.4.78.126:36575)
0924 15:23:57.676317 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.676320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.676534 (+ 214us) server_negotiation.cc:407] Connection header received
0924 15:23:57.677237 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.677238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.677248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.677290 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:57.678813 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.679549 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.680659 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.680875 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.680933 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.683325 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:57.683327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:57.683327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:57.683335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:57.684767 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.685192 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.685193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.685194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.685259 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.685551 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.685552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.685552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.685675 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:57.685751 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.686142 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.686226 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 15:23:57.686504 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.676196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:36575 (local address 127.4.78.66:38613)
0924 15:23:57.676383 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.676435 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:57.677146 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:57.677579 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:57.677583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:57.678443 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:57.678722 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.678728 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.679643 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.679644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.680559 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.680561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.680953 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.680955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.681131 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.681613 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:57.681623 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:57.683210 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:57.684871 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:57.684878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:57.684881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:57.685111 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:57.685353 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:57.685354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:57.685354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:57.685468 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:57.685743 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:57.685750 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:57.685950 (+ 200us) client_negotiation.cc:769] Sending connection context
0924 15:23:57.686109 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 15:23:57.686275 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":107,"threads_started":1}
I20260924 15:23:57.687036 15010 heartbeater.cc:347] Connected to a master server at 127.4.78.126:36575
I20260924 15:23:57.687120 15010 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:57.687263 15010 heartbeater.cc:511] Master 127.4.78.126:36575 requested a full tablet report, sending...
I20260924 15:23:57.687605 14685 ts_manager.cc:194] Registered new tserver with Master: 658a229c789b40079910e16d5c4fc527 (127.4.78.66:44865)
I20260924 15:23:57.688032 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:38613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:23:57.704430 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36575
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:40135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:23:57.784139 15015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:23:57.784337 15015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:23:57.784359 15015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:23:57.785493 15015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:23:57.785534 15015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:23:57.785588 15015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:23:57.787002 15015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:40135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:23:57.787246 15015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:23:57.787469 15015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:23:57.792299 15021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:57.792280 15023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:23:57.792297 15020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:23:57.792624 15015 server_base.cc:1034] running on GCE node
I20260924 15:23:57.792866 15015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:23:57.793097 15015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:23:57.794240 15015 hybrid_clock.cc:648] HybridClock initialized: now 1790263437794229 us; error 29 us; skew 500 ppm
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:23:57.796643 15015 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:23:57.797098 15015 webserver.cc:460] Webserver started at http://127.4.78.67:40439/ using document root <none> and password file <none>
I20260924 15:23:57.797313 15015 fs_manager.cc:359] Metadata directory not provided
I20260924 15:23:57.797356 15015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:23:57.797499 15015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:23:57.798584 15015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "17ebc8e14b5b402eb72fdbba923de55e"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "ace872f4f03bee1a98df621a3e4746ee"
server_key_iv: "3cc63e1f4b2dc4bae87a7e0125b6683f"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.798949 15015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "17ebc8e14b5b402eb72fdbba923de55e"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "ace872f4f03bee1a98df621a3e4746ee"
server_key_iv: "3cc63e1f4b2dc4bae87a7e0125b6683f"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.800274 15015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:23:57.801009 15030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.801198 15015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:23:57.801280 15015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "17ebc8e14b5b402eb72fdbba923de55e"
format_stamp: "Formatted at 2026-09-24 15:23:57 on dist-test-slave-l766"
server_key: "ace872f4f03bee1a98df621a3e4746ee"
server_key_iv: "3cc63e1f4b2dc4bae87a7e0125b6683f"
server_key_version: "encryptionkey@0"
I20260924 15:23:57.801335 15015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:23:57.818351 15015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:23:57.820456 15015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:23:57.820616 15015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:23:57.820854 15015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:23:57.821249 15015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:23:57.821286 15015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.821308 15015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:23:57.821321 15015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:23:57.827158 15015 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:41001
I20260924 15:23:57.827250 15143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:41001 every 8 connection(s)
I20260924 15:23:57.827548 15015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:23:57.828739 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 15015
I20260924 15:23:57.828816 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:23:57.828998 4409 external_mini_cluster.cc:1428] Setting key 86c258deda11c430b2f54830146d6cc4
Sep 24 15:23:57 dist-test-slave-l766 krb5kdc[14640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263437, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:23:57.836536 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.828231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:50601 (local address 127.4.78.126:36575)
0924 15:23:57.828304 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.828306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.828488 (+ 182us) server_negotiation.cc:407] Connection header received
0924 15:23:57.829078 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.829079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.829092 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.829147 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:57.830318 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.830830 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.831831 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.832044 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.832129 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.834185 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:57.834186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:57.834186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:57.834193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:57.835355 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.835738 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.835739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.835740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.835791 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.836020 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.836021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.836021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.836119 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:57.836169 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.836409 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.836472 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:23:57.836695 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.828210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:36575 (local address 127.4.78.67:50601)
0924 15:23:57.828367 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.828406 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:57.828957 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:57.829228 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:57.829233 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:57.830018 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:23:57.830215 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.830220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.830928 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.830929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.831729 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.831731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.832138 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.832139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.832276 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.832771 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:23:57.832778 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:23:57.834099 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:23:57.835427 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:57.835433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:57.835435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:57.835651 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:57.835860 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:23:57.835861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:23:57.835861 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:23:57.835962 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:23:57.836175 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:23:57.836179 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:23:57.836300 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 15:23:57.836382 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:23:57.836498 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":91,"threads_started":1}
I20260924 15:23:57.837136 15144 heartbeater.cc:347] Connected to a master server at 127.4.78.126:36575
I20260924 15:23:57.837271 15144 heartbeater.cc:464] Registering TS with master...
I20260924 15:23:57.837463 15144 heartbeater.cc:511] Master 127.4.78.126:36575 requested a full tablet report, sending...
I20260924 15:23:57.837881 14685 ts_manager.cc:194] Registered new tserver with Master: 17ebc8e14b5b402eb72fdbba923de55e (127.4.78.67:41001)
I20260924 15:23:57.838207 14685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:50601
I20260924 15:23:57.840553 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:23:57.847079 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.841731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58034 (local address 127.4.78.126:36575)
0924 15:23:57.841792 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.841794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.841876 (+ 82us) server_negotiation.cc:407] Connection header received
0924 15:23:57.841950 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.841950 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.841958 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.842002 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:23:57.842485 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.842927 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.843672 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.843847 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.843894 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.844919 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:23:57.844921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:23:57.844921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:23:57.844926 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:23:57.846140 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.846431 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.846431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.846432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.846478 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:23:57.846635 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:23:57.846636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:23:57.846636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:23:57.846767 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:23:57.846839 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.846927 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.846983 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:23:57.848686 14685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:23:57.850317 14685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:23:57.856628 15155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.852423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:44865 (local address 127.0.0.1:54170)
0924 15:23:57.852713 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.852725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:57.852777 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:57.853021 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:57.853024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:57.853026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:57.853088 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.853090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.854218 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.854219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.855653 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.855655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.856314 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.856315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.856393 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.856407 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:23:57.856495 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:23:57.856555 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":65,"threads_started":1}
I20260924 15:23:57.856640 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.852527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.4.78.66:44865)
0924 15:23:57.852833 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.852838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.852848 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:57.852893 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.852895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.852910 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.853007 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:57.853172 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.854122 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.855734 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.856227 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.856316 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.856378 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.856524 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.856565 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":87,"threads_started":1}
I20260924 15:23:57.856779 15152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.852422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:41001 (local address 127.0.0.1:51826)
0924 15:23:57.852521 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.852556 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:57.852640 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:57.853146 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:57.853147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:57.853149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:57.853207 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.853208 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.854528 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.854528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.855610 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.855612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.856494 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.856496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.856565 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.856572 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:23:57.856622 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:23:57.856681 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":11,"thread_start_us":55,"threads_started":1}
I20260924 15:23:57.857044 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.852971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.4.78.65:42335)
0924 15:23:57.853178 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.853182 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.853339 (+ 157us) server_negotiation.cc:407] Connection header received
0924 15:23:57.853382 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.853384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.853399 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.853474 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:57.853584 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.854648 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.855590 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.856325 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.856403 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.856457 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.856922 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.856969 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
I20260924 15:23:57.857065 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.852727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.4.78.67:41001)
0924 15:23:57.852914 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.852920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.852931 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:23:57.852971 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.852973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.852989 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.853070 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:57.853305 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.854420 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.855683 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.856349 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.856407 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.856457 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.856949 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.856989 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":126,"threads_started":1}
I20260924 15:23:57.857406 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.852628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:42335 (local address 127.0.0.1:43370)
0924 15:23:57.853228 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.853237 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:57.853284 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:57.853474 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:57.853475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:57.853477 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:57.853518 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.853519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.854748 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.854749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.855493 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.855495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.856495 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.856496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.856565 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.856572 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:23:57.856622 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:23:57.857340 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"mutex_wait_us":7,"thread_start_us":37,"threads_started":1}
I20260924 15:23:57.857378 14944 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4db1fee17349b480b8b244dd6ce493 (DEFAULT_TABLE table=test-table [id=9f7b932d552142d19229718e2dec149c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:57.857765 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4db1fee17349b480b8b244dd6ce493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:57.857736 15078 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4db1fee17349b480b8b244dd6ce493 (DEFAULT_TABLE table=test-table [id=9f7b932d552142d19229718e2dec149c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:57.858090 15078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4db1fee17349b480b8b244dd6ce493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:57.858067 14810 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4db1fee17349b480b8b244dd6ce493 (DEFAULT_TABLE table=test-table [id=9f7b932d552142d19229718e2dec149c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:23:57.858371 14810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4db1fee17349b480b8b244dd6ce493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:23:57.860785 15158 tablet_bootstrap.cc:492] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Bootstrap starting.
I20260924 15:23:57.861407 15158 tablet_bootstrap.cc:654] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:57.861387 15159 tablet_bootstrap.cc:492] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: Bootstrap starting.
I20260924 15:23:57.861656 15160 tablet_bootstrap.cc:492] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58: Bootstrap starting.
I20260924 15:23:57.862057 15158 log.cc:824] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:57.862129 15159 tablet_bootstrap.cc:654] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:57.862285 15160 tablet_bootstrap.cc:654] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58: Neither blocks nor log segments found. Creating new log.
I20260924 15:23:57.862731 15159 log.cc:824] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:57.862802 15160 log.cc:824] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58: Log is configured to *not* fsync() on all Append() calls
I20260924 15:23:57.862872 15158 tablet_bootstrap.cc:492] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: No bootstrap required, opened a new log
I20260924 15:23:57.862936 15158 ts_tablet_manager.cc:1397] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:57.863444 15159 tablet_bootstrap.cc:492] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: No bootstrap required, opened a new log
I20260924 15:23:57.863517 15159 ts_tablet_manager.cc:1397] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:23:57.864913 15158 raft_consensus.cc:348] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.865078 15158 raft_consensus.cc:374] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:57.865113 15158 raft_consensus.cc:729] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658a229c789b40079910e16d5c4fc527, State: Initialized, Role: FOLLOWER
I20260924 15:23:57.865252 15160 tablet_bootstrap.cc:492] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58: No bootstrap required, opened a new log
I20260924 15:23:57.865221 15158 consensus_queue.cc:260] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.865325 15160 ts_tablet_manager.cc:1397] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 15:23:57.865492 15158 ts_tablet_manager.cc:1428] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:23:57.865473 15159 raft_consensus.cc:348] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.865634 15159 raft_consensus.cc:374] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:57.865670 15159 raft_consensus.cc:729] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17ebc8e14b5b402eb72fdbba923de55e, State: Initialized, Role: FOLLOWER
I20260924 15:23:57.865667 15010 heartbeater.cc:503] Master 127.4.78.126:36575 was elected leader, sending a full tablet report...
I20260924 15:23:57.865792 15159 consensus_queue.cc:260] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.866079 15159 ts_tablet_manager.cc:1428] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:23:57.866182 15144 heartbeater.cc:503] Master 127.4.78.126:36575 was elected leader, sending a full tablet report...
I20260924 15:23:57.867220 15160 raft_consensus.cc:348] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.867372 15160 raft_consensus.cc:374] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:23:57.867417 15160 raft_consensus.cc:729] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ab9c3fc245426e868a982ce8740e58, State: Initialized, Role: FOLLOWER
I20260924 15:23:57.867524 15160 consensus_queue.cc:260] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.867792 15160 ts_tablet_manager.cc:1428] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:23:57.867828 14876 heartbeater.cc:503] Master 127.4.78.126:36575 was elected leader, sending a full tablet report...
W20260924 15:23:57.926127 15011 tablet.cc:2369] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:23:57.946265 15164 raft_consensus.cc:482] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:23:57.946385 15164 raft_consensus.cc:504] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.946830 15164 leader_election.cc:288] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17ebc8e14b5b402eb72fdbba923de55e (127.4.78.67:41001), 11ab9c3fc245426e868a982ce8740e58 (127.4.78.65:42335)
I20260924 15:23:57.949676 15156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.947214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:34247 (local address 127.4.78.67:41001)
0924 15:23:57.947262 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.947264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.947278 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:23:57.947300 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.947301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.947310 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.947375 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:57.947543 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.948011 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.948939 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.949333 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.949421 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.949432 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.949576 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.949612 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:23:57.949676 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.947005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:41001 (local address 127.4.78.66:34247)
0924 15:23:57.947076 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.947091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:57.947142 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:57.947382 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:57.947383 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:57.947385 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:57.947469 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.947471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.948106 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.948107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.948842 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.948844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.949418 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.949419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.949501 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.949504 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:57.949568 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:23:57.949606 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 15:23:57.949785 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.947005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:42335 (local address 127.4.78.66:37587)
0924 15:23:57.947219 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.947229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:23:57.947326 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:23:57.947493 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:23:57.947494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:23:57.947495 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:23:57.947571 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.947573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.948180 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.948181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.948931 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:23:57.948933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.949499 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.949500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:23:57.949578 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.949581 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:23:57.949645 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:23:57.949685 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":22,"thread_start_us":115,"threads_started":1}
I20260924 15:23:57.949769 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.947146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:37587 (local address 127.4.78.65:42335)
0924 15:23:57.947194 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.947196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.947320 (+ 124us) server_negotiation.cc:407] Connection header received
0924 15:23:57.947394 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.947395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.947407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.947487 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:23:57.947641 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.948094 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.949025 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.949411 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.949498 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.949507 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.949664 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.949701 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:23:57.950179 14830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4db1fee17349b480b8b244dd6ce493" candidate_uuid: "658a229c789b40079910e16d5c4fc527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ab9c3fc245426e868a982ce8740e58" is_pre_election: true
I20260924 15:23:57.950182 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4db1fee17349b480b8b244dd6ce493" candidate_uuid: "658a229c789b40079910e16d5c4fc527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ebc8e14b5b402eb72fdbba923de55e" is_pre_election: true
I20260924 15:23:57.950333 15098 raft_consensus.cc:2454] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658a229c789b40079910e16d5c4fc527 in term 0.
I20260924 15:23:57.950333 14830 raft_consensus.cc:2454] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658a229c789b40079910e16d5c4fc527 in term 0.
I20260924 15:23:57.950548 14899 leader_election.cc:302] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17ebc8e14b5b402eb72fdbba923de55e, 658a229c789b40079910e16d5c4fc527; no voters:
I20260924 15:23:57.950692 15164 raft_consensus.cc:2781] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:23:57.950752 15164 raft_consensus.cc:482] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:23:57.950800 15164 raft_consensus.cc:3037] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:57.951619 15164 raft_consensus.cc:504] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.951763 15164 leader_election.cc:288] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [CANDIDATE]: Term 1 election: Requested vote from peers 17ebc8e14b5b402eb72fdbba923de55e (127.4.78.67:41001), 11ab9c3fc245426e868a982ce8740e58 (127.4.78.65:42335)
I20260924 15:23:57.951929 15098 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4db1fee17349b480b8b244dd6ce493" candidate_uuid: "658a229c789b40079910e16d5c4fc527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17ebc8e14b5b402eb72fdbba923de55e"
I20260924 15:23:57.951973 14830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4db1fee17349b480b8b244dd6ce493" candidate_uuid: "658a229c789b40079910e16d5c4fc527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "11ab9c3fc245426e868a982ce8740e58"
I20260924 15:23:57.952023 15098 raft_consensus.cc:3037] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:57.952064 14830 raft_consensus.cc:3037] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:23:57.952813 14830 raft_consensus.cc:2454] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658a229c789b40079910e16d5c4fc527 in term 1.
I20260924 15:23:57.952823 15098 raft_consensus.cc:2454] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658a229c789b40079910e16d5c4fc527 in term 1.
I20260924 15:23:57.953007 14897 leader_election.cc:302] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 11ab9c3fc245426e868a982ce8740e58, 658a229c789b40079910e16d5c4fc527; no voters:
I20260924 15:23:57.953122 15164 raft_consensus.cc:2781] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:23:57.953262 15164 raft_consensus.cc:686] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 1 LEADER]: Becoming Leader. State: Replica: 658a229c789b40079910e16d5c4fc527, State: Running, Role: LEADER
I20260924 15:23:57.953373 15164 consensus_queue.cc:237] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } }
I20260924 15:23:57.954202 14684 catalog_manager.cc:5495] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 reported cstate change: term changed from 0 to 1, leader changed from <none> to 658a229c789b40079910e16d5c4fc527 (127.4.78.66). New cstate: current_term: 1 leader_uuid: "658a229c789b40079910e16d5c4fc527" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658a229c789b40079910e16d5c4fc527" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 } health_report { overall_health: UNKNOWN } } }
I20260924 15:23:57.973341 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.971268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54178 (local address 127.4.78.66:44865)
0924 15:23:57.971325 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:23:57.971327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:23:57.971337 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:23:57.971370 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:23:57.971371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:23:57.971386 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:23:57.971445 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:23:57.971575 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.972035 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.972717 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:23:57.972891 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:23:57.972937 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:23:57.973069 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:23:57.973161 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:23:57.973225 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:23:57.973257 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 15:23:57.973272 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:23:57.975093 15098 raft_consensus.cc:1261] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 1 FOLLOWER]: Refusing update from remote peer 658a229c789b40079910e16d5c4fc527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:57.975116 14830 raft_consensus.cc:1261] T 1c4db1fee17349b480b8b244dd6ce493 P 11ab9c3fc245426e868a982ce8740e58 [term 1 FOLLOWER]: Refusing update from remote peer 658a229c789b40079910e16d5c4fc527: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:23:57.975453 15168 consensus_queue.cc:1035] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17ebc8e14b5b402eb72fdbba923de55e" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:57.975576 15164 consensus_queue.cc:1035] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [LEADER]: Connected to new peer: Peer: permanent_uuid: "11ab9c3fc245426e868a982ce8740e58" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 42335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:23:57.977902 15171 mvcc.cc:205] Tried to move back new op lower bound from 7332919041943429120 to 7332919041857597440. Current Snapshot: MvccSnapshot[applied={T|T < 7332919041943429120}]
I20260924 15:23:57.978528 15172 mvcc.cc:205] Tried to move back new op lower bound from 7332919041943429120 to 7332919041857597440. Current Snapshot: MvccSnapshot[applied={T|T < 7332919041943429120}]
I20260924 15:23:57.980329 14685 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:57.980422 14685 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:23:57.981178 14685 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d4e1bee00c21400bb4ec7c4a12c1d1fc: Sending DeleteTablet for 3 replicas of tablet 1c4db1fee17349b480b8b244dd6ce493
I20260924 15:23:57.981529 15078 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4db1fee17349b480b8b244dd6ce493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:57 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:51826
I20260924 15:23:57.981654 14944 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4db1fee17349b480b8b244dd6ce493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:57 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:54170
I20260924 15:23:57.981666 14810 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4db1fee17349b480b8b244dd6ce493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:23:57 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:43370
I20260924 15:23:57.981853 15180 tablet_replica.cc:330] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: stopping tablet replica
I20260924 15:23:57.981945 15180 raft_consensus.cc:2229] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:23:57.982079 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 14747
I20260924 15:23:57.982101 15180 raft_consensus.cc:2258] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:57.983042 15180 ts_tablet_manager.cc:1905] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:57.983090 15182 tablet_replica.cc:330] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: stopping tablet replica
I20260924 15:23:57.983178 15182 raft_consensus.cc:2229] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:23:57.983253 15182 raft_consensus.cc:2258] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:23:57.983616 15182 ts_tablet_manager.cc:1905] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:23:57.985205 15182 ts_tablet_manager.cc:1918] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:57.985280 15182 log.cc:1196] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/1c4db1fee17349b480b8b244dd6ce493
I20260924 15:23:57.985271 15180 ts_tablet_manager.cc:1918] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:23:57.985345 15180 log.cc:1196] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/1c4db1fee17349b480b8b244dd6ce493
I20260924 15:23:57.985558 15182 ts_tablet_manager.cc:1939] T 1c4db1fee17349b480b8b244dd6ce493 P 658a229c789b40079910e16d5c4fc527: Deleting consensus metadata
I20260924 15:23:57.985612 15180 ts_tablet_manager.cc:1939] T 1c4db1fee17349b480b8b244dd6ce493 P 17ebc8e14b5b402eb72fdbba923de55e: Deleting consensus metadata
I20260924 15:23:57.986032 14671 catalog_manager.cc:4841] TS 17ebc8e14b5b402eb72fdbba923de55e (127.4.78.67:41001): tablet 1c4db1fee17349b480b8b244dd6ce493 (table test-table [id=9f7b932d552142d19229718e2dec149c]) successfully deleted
I20260924 15:23:57.986088 14669 catalog_manager.cc:4841] TS 658a229c789b40079910e16d5c4fc527 (127.4.78.66:44865): tablet 1c4db1fee17349b480b8b244dd6ce493 (table test-table [id=9f7b932d552142d19229718e2dec149c]) successfully deleted
W20260924 15:23:57.987529 14669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:42335 (error 108)
I20260924 15:23:57.988024 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:23:57.987892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:42335 (local address 127.0.0.1:43378)
0924 15:23:57.987953 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 15:23:57.987993 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:42335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 15:23:57.988093 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 14881
W20260924 15:23:57.988771 14669 catalog_manager.cc:4584] TS 11ab9c3fc245426e868a982ce8740e58 (127.4.78.65:42335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c4db1fee17349b480b8b244dd6ce493: Network error: Client connection negotiation failed: client connection to 127.4.78.65:42335: connect: Connection refused (error 111)
I20260924 15:23:57.993752 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 15015
I20260924 15:23:57.998567 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 14653
2026-09-24T15:23:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1951 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:23:58 dist-test-slave-l766 krb5kdc[15187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:23:58 dist-test-slave-l766 krb5kdc[15187](info): set up 2 sockets
Sep 24 15:23:58 dist-test-slave-l766 krb5kdc[15187](info): commencing operation
krb5kdc: starting...
W20260924 15:23:59.041782 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:23:59 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:23:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:23:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.78.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.78.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/rangeradmin_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/rangeradmin_127.4.78.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.78.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.78.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/rangerlookup_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/rangerlookup_127.4.78.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.78.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.78.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/HTTP_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/HTTP_127.4.78.84@KRBTEST.COM.keytab.
I20260924 15:23:59.198653 4409 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwmKTEk/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/postgres -l logfile start
2026-09-24 15:24:01.598 UTC [15218] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 15:24:01.598 UTC [15218] LOG: listening on IPv4 address "127.4.78.84", port 36265
2026-09-24 15:24:01.601 UTC [15218] LOG: listening on Unix socket "/tmp/.s.PGSQL.36265"
2026-09-24 15:24:01.635 UTC [15221] LOG: database system was shut down at 2026-09-24 15:24:00 UTC
2026-09-24 15:24:01.639 UTC [15218] LOG: database system is ready to accept connections
I20260924 15:24:02.581552 4409 mini_postgres.cc:96] Postgres bound to 36265
2026-09-24 15:24:02.587 UTC [15229] FATAL: database "slave" does not exist
127.4.78.84:36265 - accepting connections
I20260924 15:24:02.588066 4409 mini_ranger.cc:161] Starting Ranger...
I20260924 15:24:02.602121 4409 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 15:24:03.056880 4409 mini_ranger.cc:87] Created ranger Postgres database
I20260924 15:24:03.056967 4409 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-admin
2026-09-24 15:24:03,343 [I] DB FLAVOR :POSTGRES
2026-09-24 15:24:03,343 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:24:03,344 [I] Checking connection..
2026-09-24 15:24:03,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 15:24:03,601 [I] Checking connection passed.
2026-09-24 15:24:03,601 [I] --------- Verifying version history table ---------
2026-09-24 15:24:03,601 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:24:03,843 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:24:03,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:24:04,099 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:24:04,099 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 15:24:04,099 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 15:24:04.347 UTC [15295] WARNING: there is no transaction in progress
2026-09-24 15:24:04,353 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 15:24:04,353 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:24:04,580 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 15:24:04,580 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:24:04,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 15:24:04,817 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 15:24:05,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l766.c.gcp-upstream.internal','N') ;"
2026-09-24 15:24:05,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:24:05,526 [I] Table x_portal_user does not exist in database ranger
2026-09-24 15:24:05,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:24:05,775 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 15:24:05,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:24:05,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:24:06,226 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 15:24:06,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 15:24:07.155 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.171 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.186 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.200 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.218 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.238 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.302 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.308 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.322 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.331 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.341 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.354 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.363 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.369 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.375 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07.381 UTC [15404] WARNING: there is no transaction in progress
2026-09-24 15:24:07,979 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 15:24:07,980 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 15:24:08,234 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 15:24:08,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:24:08,486 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 15:24:08,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:24:08,742 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 15:24:08,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:24:08,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:24:09,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l766.c.gcp-upstream.internal';"
2026-09-24 15:24:09,475 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 15:24:09,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 15:24:09,714 [I] DB_PATCHES have already been applied
I20260924 15:24:09.720250 4409 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwmKTEk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 15:24:09.720312 4409 mini_ranger.cc:193] Using host: 127.4.78.84
I20260924 15:24:09.723376 4409 mini_ranger.cc:227] Ranger admin URL: http://127.4.78.84:46099
Sep 24, 2026 3:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:24:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 15:24:10.841689 4409 mini_ranger.cc:160] Time spent starting Ranger: real 8.254s user 0.000s sys 0.009s
Sep 24, 2026 3:24:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:24:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:24:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:24:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:24:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 15:24:32.976756 4409 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.78.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.78.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/rangerkms_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/rangerkms_127.4.78.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.78.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.78.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.78.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/HTTP_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.78.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/HTTP_127.4.78.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 15:24:33 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263473, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 15:24:33.062624 4409 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 15:24:33.072026 4409 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 15:24:34.047216 4409 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 15:24:34.047294 4409 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms
I20260924 15:24:34.047308 4409 mini_ranger_kms.cc:227] Using postgres at 127.4.78.84:36265
2026-09-24 15:24:34,671 [I] DB FLAVOR :POSTGRES
2026-09-24 15:24:34,671 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:24:34,672 [I] Checking connection
2026-09-24 15:24:34,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 15:24:34,904 [I] connection success
2026-09-24 15:24:34,904 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 15:24:34,904 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 15:24:34,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 15:24:35,164 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 15:24:35,164 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:24:35,164 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 15:24:35,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:36265/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 15:24:35,430 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 15:24:35.941040 4409 mini_ranger_kms.cc:310] Created kms service
I20260924 15:24:36.125082 4409 mini_ranger_kms.cc:326] Created kudu user
I20260924 15:24:36.171823 4409 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 15:24:36 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263473, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:24:36.541627 4409 mini_ranger_kms.cc:384] Added ranger policy
I20260924 15:24:36.541755 4409 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 15:24:36.541779 4409 mini_ranger_kms.cc:240] Using host: 127.4.78.84
I20260924 15:24:36.545519 4409 mini_ranger_kms.cc:276] Ranger KMS PID: 15585
I20260924 15:24:36.545637 4409 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.78.84:36271
Sep 24, 2026 3:24:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:24:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:24:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:24:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:24:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:24:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 15:24:38.814033 4409 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.751s user 0.000s sys 0.229s
I20260924 15:24:38.814148 4409 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 15:24:38.814244 4409 mini_ranger_kms.cc:401] 127.4.78.84:36271/kms/v1/keys
Sep 24 15:24:38 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.78.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 15:24:39 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.78.84@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
Sep 24 15:24:39 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263478, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.78.84@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
Sep 24 15:24:41 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:24:41.904666 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:41125
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:41125
--ranger_config_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:36271/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:24:41.992142 15637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:24:41.992347 15637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:24:41.992367 15637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:24:41.993541 15637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:24:41.993582 15637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:24:41.993597 15637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:24:41.993652 15637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:24:41.993677 15637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:24:41.995263 15637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.78.84:36271/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:41125
--ranger_config_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:41125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:24:41.995532 15637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:24:41.995779 15637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:24:42.001235 15637 server_base.cc:1034] running on GCE node
W20260924 15:24:42.001331 15643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:24:42.001197 15645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:24:42.001199 15642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:24:42.001646 15637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:24:42.002012 15637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:24:42.003185 15637 hybrid_clock.cc:648] HybridClock initialized: now 1790263482003177 us; error 34 us; skew 500 ppm
Sep 24 15:24:41 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263481, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:24:42.005689 15637 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:24:42.006278 15637 webserver.cc:460] Webserver started at http://127.4.78.126:33913/ using document root <none> and password file <none>
I20260924 15:24:42.006500 15637 fs_manager.cc:359] Metadata directory not provided
I20260924 15:24:42.006559 15637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:24:42.006697 15637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:24:42 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263481, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:24:42.069053 15637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "9fa4f847d03b4c10aa79d7bba1e1642a"
format_stamp: "Formatted at 2026-09-24 15:24:42 on dist-test-slave-l766"
server_key: "1f6f22c67e31112ebf70fd1bfff56558"
server_key_iv: "507fd4a3c69fee2e2f02054588b5d4a8"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:42.069483 15637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "9fa4f847d03b4c10aa79d7bba1e1642a"
format_stamp: "Formatted at 2026-09-24 15:24:42 on dist-test-slave-l766"
server_key: "1f6f22c67e31112ebf70fd1bfff56558"
server_key_iv: "507fd4a3c69fee2e2f02054588b5d4a8"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:42.071062 15637 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 15:24:42.082252 15652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:42.082453 15637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:24:42.082571 15637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "9fa4f847d03b4c10aa79d7bba1e1642a"
format_stamp: "Formatted at 2026-09-24 15:24:42 on dist-test-slave-l766"
server_key: "1f6f22c67e31112ebf70fd1bfff56558"
server_key_iv: "507fd4a3c69fee2e2f02054588b5d4a8"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:42.082682 15637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:24:42.092305 15637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:24:42.094493 15637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:24:42.094661 15637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:24:42.099483 15637 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:41125
I20260924 15:24:42.099520 15704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:41125 every 8 connection(s)
I20260924 15:24:42.099941 15637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:24:42.100654 15705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:24:42.100718 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 15637
I20260924 15:24:42.100823 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:24:42.101011 4409 external_mini_cluster.cc:1428] Setting key 354508ec541b3b04955ad731d5df4f72
I20260924 15:24:42.103708 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a: Bootstrap starting.
I20260924 15:24:42.104326 15705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a: Neither blocks nor log segments found. Creating new log.
I20260924 15:24:42.104619 15705 log.cc:824] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:24:42.105391 15705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a: No bootstrap required, opened a new log
I20260924 15:24:42.107031 15705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa4f847d03b4c10aa79d7bba1e1642a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41125 } }
I20260924 15:24:42.107194 15705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:24:42.107257 15705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa4f847d03b4c10aa79d7bba1e1642a, State: Initialized, Role: FOLLOWER
I20260924 15:24:42.107494 15705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa4f847d03b4c10aa79d7bba1e1642a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41125 } }
I20260924 15:24:42.107688 15705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:24:42.107750 15705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:24:42.107795 15705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:24:42.108525 15705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa4f847d03b4c10aa79d7bba1e1642a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41125 } }
I20260924 15:24:42.108647 15705 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa4f847d03b4c10aa79d7bba1e1642a; no voters:
I20260924 15:24:42.108829 15705 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:24:42.108884 15710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:24:42.109156 15710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [term 1 LEADER]: Becoming Leader. State: Replica: 9fa4f847d03b4c10aa79d7bba1e1642a, State: Running, Role: LEADER
I20260924 15:24:42.109227 15705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:24:42.109314 15710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa4f847d03b4c10aa79d7bba1e1642a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41125 } }
Sep 24 15:24:42 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:24:42.109983 15710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa4f847d03b4c10aa79d7bba1e1642a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa4f847d03b4c10aa79d7bba1e1642a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41125 } } }
I20260924 15:24:42.110054 15710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [sys.catalog]: This master's current role is: LEADER
I20260924 15:24:42.110342 15710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa4f847d03b4c10aa79d7bba1e1642a. Latest consensus state: current_term: 1 leader_uuid: "9fa4f847d03b4c10aa79d7bba1e1642a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa4f847d03b4c10aa79d7bba1e1642a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41125 } } }
I20260924 15:24:42.110448 15710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a [sys.catalog]: This master's current role is: LEADER
I20260924 15:24:42.110595 15719 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:24:42.110853 15705 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:24:42.111383 15719 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:24:42.117862 15728 mvcc.cc:205] Tried to move back new op lower bound from 7332919222748164096 to 7332919222721417216. Current Snapshot: MvccSnapshot[applied={T|T < 7332919222748164096}]
I20260924 15:24:42.119540 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:42.101778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51370 (local address 127.4.78.126:41125)
0924 15:24:42.102006 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:42.102016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:42.102051 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:24:42.102598 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:42.102607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:42.102674 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:42.102823 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:24:42.103595 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:42.105044 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:42.106125 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:42.107286 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:42.107638 (+ 352us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:42.110678 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:24:42.110682 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:24:42.110682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:24:42.110694 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:24:42.116858 (+ 6164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:42.117375 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:42.117379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:42.117379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:42.117504 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:42.117761 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:42.117762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:42.117762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:42.118196 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:24:42.118370 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:42.119126 (+ 756us) server_negotiation.cc:299] Negotiation successful
0924 15:24:42.119306 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":120,"threads_started":1}
I20260924 15:24:42.122772 15719 catalog_manager.cc:1347] Generated new cluster ID: 691160b50045412fb58eec74984d041f
I20260924 15:24:42.122889 15719 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:24:42.145073 15719 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:24:42.146090 15719 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:24:42.149478 15719 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a: Generated new TSK 0
I20260924 15:24:42.149744 15719 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:24:42.886943 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:36271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:41125
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:24:43.021281 15750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:24:43.021550 15750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:24:43.021588 15750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:24:43.023272 15750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:24:43.023337 15750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:24:43.023375 15750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:24:43.025520 15750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.78.84:36271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:24:43.025760 15750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:24:43.026068 15750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:24:43.031548 15755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:24:43.031559 15756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:24:43.031744 15750 server_base.cc:1034] running on GCE node
W20260924 15:24:43.031589 15758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:24:43.032089 15750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:24:43.032363 15750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:24:43.033550 15750 hybrid_clock.cc:648] HybridClock initialized: now 1790263483033536 us; error 26 us; skew 500 ppm
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:24:43.036422 15750 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:24:43.036871 15750 webserver.cc:460] Webserver started at http://127.4.78.65:41021/ using document root <none> and password file <none>
I20260924 15:24:43.037092 15750 fs_manager.cc:359] Metadata directory not provided
I20260924 15:24:43.037158 15750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:24:43.037292 15750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:24:43.051397 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "7e470510cfd74825a22c0f434559b021"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "28eca6684bae73746765edaac1f72cb2"
server_key_iv: "b0b308029ff6f4120327b74242b6b69f"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.051942 15750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "7e470510cfd74825a22c0f434559b021"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "28eca6684bae73746765edaac1f72cb2"
server_key_iv: "b0b308029ff6f4120327b74242b6b69f"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.053934 15750 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:24:43.065699 15765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.065924 15750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:24:43.066021 15750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "7e470510cfd74825a22c0f434559b021"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "28eca6684bae73746765edaac1f72cb2"
server_key_iv: "b0b308029ff6f4120327b74242b6b69f"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.066133 15750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:24:43.101701 15750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:24:43.104355 15750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:24:43.104568 15750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:24:43.104912 15750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:24:43.105343 15750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:24:43.105398 15750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.105466 15750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:24:43.105510 15750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.116766 15750 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:45117
I20260924 15:24:43.116878 15881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:45117 every 8 connection(s)
I20260924 15:24:43.117233 15750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:24:43.125254 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 15750
I20260924 15:24:43.125370 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:24:43.125568 4409 external_mini_cluster.cc:1428] Setting key 02c68c426184595e4d4fc780ebdd0698
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:24:43.130671 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.118231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:34143 (local address 127.4.78.126:41125)
0924 15:24:43.121940 (+ 3709us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.121942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.121951 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:24:43.121974 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.121975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.121986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.122081 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:24:43.122688 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.123204 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.124218 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.124437 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.124521 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.127144 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:24:43.127145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:24:43.127145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:24:43.127152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:24:43.129147 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:43.129599 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:43.129601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:43.129601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:43.129663 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:43.129996 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:43.129997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:43.129997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:43.130119 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:24:43.130195 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.130487 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.130577 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3652,"thread_start_us":79,"threads_started":1}
I20260924 15:24:43.130836 15884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.118190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:41125 (local address 127.4.78.65:34143)
0924 15:24:43.118428 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.118470 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:24:43.119159 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:24:43.122188 (+ 3029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:24:43.122192 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:24:43.122381 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:24:43.122577 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.122581 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.123308 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.123309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.124112 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.124114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.124530 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.124532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.124658 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.125236 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:24:43.125248 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:24:43.127045 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:24:43.129260 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:24:43.129266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:24:43.129269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:24:43.129519 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:24:43.129756 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:24:43.129757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:24:43.129757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:24:43.129927 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:24:43.130187 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:24:43.130192 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:24:43.130350 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 15:24:43.130455 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:24:43.130597 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":98,"threads_started":1}
I20260924 15:24:43.131285 15882 heartbeater.cc:347] Connected to a master server at 127.4.78.126:41125
I20260924 15:24:43.131381 15882 heartbeater.cc:464] Registering TS with master...
I20260924 15:24:43.131556 15882 heartbeater.cc:511] Master 127.4.78.126:41125 requested a full tablet report, sending...
I20260924 15:24:43.132083 15669 ts_manager.cc:194] Registered new tserver with Master: 7e470510cfd74825a22c0f434559b021 (127.4.78.65:45117)
I20260924 15:24:43.132669 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:34143
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:24:43.157701 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:36271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:41125
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:24:43.277086 15888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:24:43.277405 15888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:24:43.277463 15888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 15:24:43.278816 15888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:24:43.278885 15888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:24:43.278980 15888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:24:43.280587 15888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.78.84:36271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:24:43.280843 15888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:24:43.281076 15888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:24:43.288147 15895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:24:43.288131 15898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:24:43.288582 15896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:24:43.289395 15888 server_base.cc:1034] running on GCE node
I20260924 15:24:43.289763 15888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:24:43.290115 15888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:24:43.293406 15888 hybrid_clock.cc:648] HybridClock initialized: now 1790263483293357 us; error 60 us; skew 500 ppm
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:24:43.297163 15888 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:24:43.297756 15888 webserver.cc:460] Webserver started at http://127.4.78.66:38027/ using document root <none> and password file <none>
I20260924 15:24:43.298095 15888 fs_manager.cc:359] Metadata directory not provided
I20260924 15:24:43.298207 15888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:24:43.298396 15888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:24:43.317699 15888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "79752c65cf254ddab0e3e48e49d2beeb"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "7a5e5a35794392f94af1b9b3484e689e"
server_key_iv: "f3c5198dc94e5edfd639ee3f8e722f36"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.318225 15888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "79752c65cf254ddab0e3e48e49d2beeb"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "7a5e5a35794392f94af1b9b3484e689e"
server_key_iv: "f3c5198dc94e5edfd639ee3f8e722f36"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.320257 15888 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 15:24:43.333025 15905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.333277 15888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:24:43.333396 15888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "79752c65cf254ddab0e3e48e49d2beeb"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "7a5e5a35794392f94af1b9b3484e689e"
server_key_iv: "f3c5198dc94e5edfd639ee3f8e722f36"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.333491 15888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:24:43.348618 15888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:24:43.351279 15888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:24:43.351451 15888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:24:43.351715 15888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:24:43.352094 15888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:24:43.352139 15888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.352170 15888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:24:43.352188 15888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.364729 15888 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:44831
I20260924 15:24:43.365221 15888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:24:43.365765 16020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:44831 every 8 connection(s)
I20260924 15:24:43.366775 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 15888
I20260924 15:24:43.366878 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:24:43.367056 4409 external_mini_cluster.cc:1428] Setting key 5074701f5369b8d360db9399626442b4
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:24:43.377027 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.366068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:41125 (local address 127.4.78.66:51025)
0924 15:24:43.366367 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.366413 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:24:43.367059 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:24:43.367440 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:24:43.367444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:24:43.367690 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:24:43.367962 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.367968 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.368962 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.368963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.370310 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.370313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.370884 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.370886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.371042 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.371644 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:24:43.371655 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:24:43.373368 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:24:43.375386 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:24:43.375393 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:24:43.375396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:24:43.375698 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:24:43.375905 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:24:43.375905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:24:43.375906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:24:43.376026 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:24:43.376335 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:24:43.376341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:24:43.376518 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 15:24:43.376628 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:24:43.376788 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":89,"threads_started":1}
I20260924 15:24:43.377463 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.366056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:51025 (local address 127.4.78.126:41125)
0924 15:24:43.366120 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.366123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.366530 (+ 407us) server_negotiation.cc:407] Connection header received
0924 15:24:43.367204 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.367205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.367217 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.367249 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:24:43.368070 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.368846 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.370406 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.370729 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.370810 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.373476 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:24:43.373477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:24:43.373478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:24:43.373485 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:24:43.375292 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:43.375781 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:43.375782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:43.375783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:43.375841 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:43.376093 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:43.376094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:43.376094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:43.376237 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:24:43.376300 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.377283 (+ 983us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.377373 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:24:43.377847 16021 heartbeater.cc:347] Connected to a master server at 127.4.78.126:41125
I20260924 15:24:43.377952 16021 heartbeater.cc:464] Registering TS with master...
I20260924 15:24:43.378144 16021 heartbeater.cc:511] Master 127.4.78.126:41125 requested a full tablet report, sending...
I20260924 15:24:43.378505 15669 ts_manager.cc:194] Registered new tserver with Master: 79752c65cf254ddab0e3e48e49d2beeb (127.4.78.66:44831)
I20260924 15:24:43.379007 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:51025
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:24:43.408865 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:36271/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:41125
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:24:43.525278 16027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:24:43.525615 16027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:24:43.525705 16027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:24:43.527432 16027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:24:43.527546 16027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:24:43.527626 16027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:24:43.529794 16027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.78.84:36271/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:41125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:24:43.530354 16027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:24:43.530674 16027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:24:43.537297 16032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:24:43.538028 16033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:24:43.538275 16027 server_base.cc:1034] running on GCE node
W20260924 15:24:43.547009 16035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:24:43.547317 16027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:24:43.547623 16027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:24:43.554963 16027 hybrid_clock.cc:648] HybridClock initialized: now 1790263483554938 us; error 35 us; skew 500 ppm
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:24:43.558470 16027 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:24:43.559000 16027 webserver.cc:460] Webserver started at http://127.4.78.67:45807/ using document root <none> and password file <none>
I20260924 15:24:43.559248 16027 fs_manager.cc:359] Metadata directory not provided
I20260924 15:24:43.559299 16027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:24:43.559422 16027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:24:43.583043 16027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "c629cfca60004c6eae9ea999f758b932"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "977d76eadd894038c4b9296ca1441bc2"
server_key_iv: "94f5b29866d3f47dca32bec3668c7bc3"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.583542 16027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "c629cfca60004c6eae9ea999f758b932"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "977d76eadd894038c4b9296ca1441bc2"
server_key_iv: "94f5b29866d3f47dca32bec3668c7bc3"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.585309 16027 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 15:24:43.605465 16042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.605746 16027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 15:24:43.605922 16027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "c629cfca60004c6eae9ea999f758b932"
format_stamp: "Formatted at 2026-09-24 15:24:43 on dist-test-slave-l766"
server_key: "977d76eadd894038c4b9296ca1441bc2"
server_key_iv: "94f5b29866d3f47dca32bec3668c7bc3"
server_key_version: "kuduclusterkey@0"
I20260924 15:24:43.606051 16027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:24:43.644966 16027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:24:43.647518 16027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:24:43.647666 16027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:24:43.647924 16027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:24:43.648283 16027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:24:43.648329 16027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.648363 16027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:24:43.648386 16027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:24:43.655439 16027 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:46835
I20260924 15:24:43.656006 16027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:24:43.656157 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 16027
I20260924 15:24:43.656250 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:24:43.656447 4409 external_mini_cluster.cc:1428] Setting key bd575cc0f7a36a12ee9303468b6e31e8
I20260924 15:24:43.667297 16156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:46835 every 8 connection(s)
Sep 24 15:24:43 dist-test-slave-l766 krb5kdc[15187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263483, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:24:43.679852 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.668914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:35701 (local address 127.4.78.126:41125)
0924 15:24:43.668954 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.668956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.669159 (+ 203us) server_negotiation.cc:407] Connection header received
0924 15:24:43.669870 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.669872 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.669884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.669914 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:24:43.670773 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.671547 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.673045 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.673365 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.673409 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.676421 (+ 3012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:24:43.676423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:24:43.676423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:24:43.676430 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:24:43.678148 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:43.678646 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:43.678648 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:43.678648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:43.678708 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:43.679017 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:43.679018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:43.679018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:43.679149 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:24:43.679191 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.679658 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.679755 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:24:43.680410 16159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.668648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:41125 (local address 127.4.78.67:35701)
0924 15:24:43.669021 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.669064 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:24:43.669723 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:24:43.670101 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:24:43.670105 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:24:43.670366 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:24:43.670664 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.670671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.671665 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.671666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.672943 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.672946 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.673501 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.673502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.673643 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.674723 (+ 1080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:24:43.674734 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:24:43.676313 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:24:43.678246 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:24:43.678253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:24:43.678255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:24:43.678547 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:24:43.678800 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:24:43.678801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:24:43.678801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:24:43.678932 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:24:43.679244 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:24:43.679250 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:24:43.679486 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 15:24:43.679945 (+ 459us) client_negotiation.cc:240] Negotiation successful
0924 15:24:43.680101 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":96,"threads_started":1}
I20260924 15:24:43.681008 16157 heartbeater.cc:347] Connected to a master server at 127.4.78.126:41125
I20260924 15:24:43.681100 16157 heartbeater.cc:464] Registering TS with master...
I20260924 15:24:43.681274 16157 heartbeater.cc:511] Master 127.4.78.126:41125 requested a full tablet report, sending...
I20260924 15:24:43.681602 15669 ts_manager.cc:194] Registered new tserver with Master: c629cfca60004c6eae9ea999f758b932 (127.4.78.67:46835)
I20260924 15:24:43.682081 15669 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:35701
I20260924 15:24:43.689284 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:24:43.698302 15885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.690943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.4.78.126:41125)
0924 15:24:43.690979 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.690981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.691215 (+ 234us) server_negotiation.cc:407] Connection header received
0924 15:24:43.691303 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.691304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.691314 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.691341 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:24:43.692052 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.692765 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.693779 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.694068 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.694107 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.695567 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:24:43.695569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:24:43.695569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:24:43.695576 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:24:43.697127 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:43.697490 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:43.697491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:43.697492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:43.697539 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:24:43.697737 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:24:43.697738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:24:43.697738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:24:43.697870 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:24:43.697940 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.698143 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.698216 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:24:43.700296 15669 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:24:43.702623 15669 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:24:43.712347 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.705695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:44831 (local address 127.0.0.1:48814)
0924 15:24:43.706258 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.706680 (+ 422us) client_negotiation.cc:174] Beginning negotiation
0924 15:24:43.706835 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:24:43.707320 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:24:43.707323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:24:43.707325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:24:43.707392 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.707394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.709055 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.709056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.710977 (+ 1921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.710979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.712043 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.712044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.712146 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.712163 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 15:24:43.712211 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:24:43.712261 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":84,"threads_started":1}
I20260924 15:24:43.712741 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.706074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40868 (local address 127.4.78.65:45117)
0924 15:24:43.706620 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.706625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.707070 (+ 445us) server_negotiation.cc:407] Connection header received
0924 15:24:43.707129 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.707132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.707152 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.707246 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:24:43.707502 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.708616 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.711166 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.711925 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.711999 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.712153 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.712404 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.712666 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":65,"threads_started":1}
I20260924 15:24:43.713021 16173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.705956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:45117 (local address 127.0.0.1:40868)
0924 15:24:43.706965 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.706974 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:24:43.707022 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:24:43.707330 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:24:43.707331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:24:43.707333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:24:43.707419 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.707421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.709923 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.709924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.711056 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.711059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.712190 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.712191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.712267 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.712271 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:24:43.712324 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:24:43.712362 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"thread_start_us":160,"threads_started":1}
I20260924 15:24:43.712898 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.706339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48814 (local address 127.4.78.66:44831)
0924 15:24:43.707136 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.707140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.707150 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:24:43.707189 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.707191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.707206 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.707465 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:24:43.707537 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.708969 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.711137 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.711945 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.712688 (+ 743us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.712768 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.712793 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.712828 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":99,"threads_started":1}
I20260924 15:24:43.713768 15816 tablet_service.cc:1467] Processing CreateTablet for tablet 14c0650af57241b795c84505dd4e9665 (DEFAULT_TABLE table=test-table [id=d6eea320849a48faaed59d49c2b845d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:24:43.714035 15954 tablet_service.cc:1467] Processing CreateTablet for tablet 14c0650af57241b795c84505dd4e9665 (DEFAULT_TABLE table=test-table [id=d6eea320849a48faaed59d49c2b845d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:24:43.714202 15816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c0650af57241b795c84505dd4e9665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:24:43.714387 15954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c0650af57241b795c84505dd4e9665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:24:43.718304 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.706208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:46835 (local address 127.0.0.1:49518)
0924 15:24:43.712438 (+ 6230us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.712448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:24:43.712487 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:24:43.714009 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:24:43.714010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:24:43.714012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:24:43.714043 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.714044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.715451 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.715452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.716499 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.716502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.718115 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.718116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.718175 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.718179 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:24:43.718206 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 15:24:43.718242 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6220,"thread_start_us":63,"threads_started":1}
I20260924 15:24:43.718721 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.706761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.4.78.67:46835)
0924 15:24:43.713786 (+ 7025us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.713791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.713803 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:24:43.713885 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.713887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.713904 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.714109 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:24:43.714231 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.715327 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.716639 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.718022 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.718491 (+ 469us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.718573 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.718598 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.718647 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6893,"thread_start_us":94,"threads_started":1}
I20260924 15:24:43.719278 16091 tablet_service.cc:1467] Processing CreateTablet for tablet 14c0650af57241b795c84505dd4e9665 (DEFAULT_TABLE table=test-table [id=d6eea320849a48faaed59d49c2b845d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:24:43.719580 16091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14c0650af57241b795c84505dd4e9665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:24:43.723104 16176 tablet_bootstrap.cc:492] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: Bootstrap starting.
I20260924 15:24:43.724143 16176 tablet_bootstrap.cc:654] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: Neither blocks nor log segments found. Creating new log.
I20260924 15:24:43.724435 16176 log.cc:824] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:24:43.726016 16177 tablet_bootstrap.cc:492] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021: Bootstrap starting.
I20260924 15:24:43.726613 16177 tablet_bootstrap.cc:654] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021: Neither blocks nor log segments found. Creating new log.
I20260924 15:24:43.726903 16177 log.cc:824] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021: Log is configured to *not* fsync() on all Append() calls
I20260924 15:24:43.727262 16178 tablet_bootstrap.cc:492] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: Bootstrap starting.
I20260924 15:24:43.728142 16178 tablet_bootstrap.cc:654] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: Neither blocks nor log segments found. Creating new log.
I20260924 15:24:43.728420 16178 log.cc:824] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: Log is configured to *not* fsync() on all Append() calls
I20260924 15:24:43.728988 16177 tablet_bootstrap.cc:492] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021: No bootstrap required, opened a new log
I20260924 15:24:43.729099 16177 ts_tablet_manager.cc:1397] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:24:43.731287 16177 raft_consensus.cc:348] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.731487 16177 raft_consensus.cc:374] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:24:43.731536 16177 raft_consensus.cc:729] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e470510cfd74825a22c0f434559b021, State: Initialized, Role: FOLLOWER
I20260924 15:24:43.731904 16178 tablet_bootstrap.cc:492] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: No bootstrap required, opened a new log
I20260924 15:24:43.731989 16178 ts_tablet_manager.cc:1397] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 15:24:43.733916 16177 consensus_queue.cc:260] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.734006 16178 raft_consensus.cc:348] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.734165 16178 raft_consensus.cc:374] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:24:43.734205 16178 raft_consensus.cc:729] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c629cfca60004c6eae9ea999f758b932, State: Initialized, Role: FOLLOWER
I20260924 15:24:43.734336 16178 consensus_queue.cc:260] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.734593 16178 ts_tablet_manager.cc:1428] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:24:43.734710 16157 heartbeater.cc:503] Master 127.4.78.126:41125 was elected leader, sending a full tablet report...
I20260924 15:24:43.734961 16177 ts_tablet_manager.cc:1428] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260924 15:24:43.737893 15882 heartbeater.cc:503] Master 127.4.78.126:41125 was elected leader, sending a full tablet report...
I20260924 15:24:43.739030 16176 tablet_bootstrap.cc:492] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: No bootstrap required, opened a new log
I20260924 15:24:43.739107 16176 ts_tablet_manager.cc:1397] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20260924 15:24:43.741014 16176 raft_consensus.cc:348] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.741183 16176 raft_consensus.cc:374] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:24:43.741223 16176 raft_consensus.cc:729] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79752c65cf254ddab0e3e48e49d2beeb, State: Initialized, Role: FOLLOWER
I20260924 15:24:43.741333 16176 consensus_queue.cc:260] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.741582 16176 ts_tablet_manager.cc:1428] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:24:43.741737 16021 heartbeater.cc:503] Master 127.4.78.126:41125 was elected leader, sending a full tablet report...
I20260924 15:24:43.759076 16184 raft_consensus.cc:482] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:24:43.759220 16184 raft_consensus.cc:504] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.763413 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.759731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:45117 (local address 127.4.78.66:54627)
0924 15:24:43.759824 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.759835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:24:43.759874 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:24:43.760180 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:24:43.760181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:24:43.760182 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:24:43.760253 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.760255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.761161 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.761162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.762396 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.762399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.763180 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.763182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.763252 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.763255 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:24:43.763298 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 15:24:43.763338 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 15:24:43.763690 16172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.759972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:54627 (local address 127.4.78.65:45117)
0924 15:24:43.760012 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.760014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.760022 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:24:43.760046 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.760047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.760057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.760130 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:24:43.760320 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.761023 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.762495 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.763060 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.763137 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.763148 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.763584 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.763628 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:24:43.764008 15836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c0650af57241b795c84505dd4e9665" candidate_uuid: "79752c65cf254ddab0e3e48e49d2beeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e470510cfd74825a22c0f434559b021" is_pre_election: true
I20260924 15:24:43.764166 15836 raft_consensus.cc:2454] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79752c65cf254ddab0e3e48e49d2beeb in term 0.
I20260924 15:24:43.764669 15906 leader_election.cc:302] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79752c65cf254ddab0e3e48e49d2beeb, 7e470510cfd74825a22c0f434559b021; no voters:
I20260924 15:24:43.768244 16023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.765035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:46835 (local address 127.4.78.66:48057)
0924 15:24:43.765200 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.765210 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:24:43.765247 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:24:43.765404 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:24:43.765405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:24:43.765406 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:24:43.765434 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.765435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.766326 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.766327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.767330 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:24:43.767333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.768036 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.768037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:24:43.768091 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.768094 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:24:43.768150 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:24:43.768182 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 15:24:43.768496 16175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.765134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:48057 (local address 127.4.78.67:46835)
0924 15:24:43.765176 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.765178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.765293 (+ 115us) server_negotiation.cc:407] Connection header received
0924 15:24:43.765317 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.765318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.765328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.765367 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:24:43.765492 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.766215 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.767426 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.767938 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.767999 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.768007 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.768398 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.768435 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:24:43.768785 16108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c0650af57241b795c84505dd4e9665" candidate_uuid: "79752c65cf254ddab0e3e48e49d2beeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c629cfca60004c6eae9ea999f758b932" is_pre_election: true
I20260924 15:24:43.768923 16108 raft_consensus.cc:2454] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79752c65cf254ddab0e3e48e49d2beeb in term 0.
I20260924 15:24:43.769182 16184 leader_election.cc:288] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e470510cfd74825a22c0f434559b021 (127.4.78.65:45117), c629cfca60004c6eae9ea999f758b932 (127.4.78.67:46835)
I20260924 15:24:43.769287 16184 raft_consensus.cc:2781] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:24:43.769332 16184 raft_consensus.cc:482] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:24:43.769361 16184 raft_consensus.cc:3037] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:24:43.770256 16184 raft_consensus.cc:504] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.770495 15836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c0650af57241b795c84505dd4e9665" candidate_uuid: "79752c65cf254ddab0e3e48e49d2beeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e470510cfd74825a22c0f434559b021"
I20260924 15:24:43.770576 15836 raft_consensus.cc:3037] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:24:43.771389 15836 raft_consensus.cc:2454] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79752c65cf254ddab0e3e48e49d2beeb in term 1.
I20260924 15:24:43.771551 15906 leader_election.cc:302] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79752c65cf254ddab0e3e48e49d2beeb, 7e470510cfd74825a22c0f434559b021; no voters:
I20260924 15:24:43.771756 16108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14c0650af57241b795c84505dd4e9665" candidate_uuid: "79752c65cf254ddab0e3e48e49d2beeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c629cfca60004c6eae9ea999f758b932"
I20260924 15:24:43.771829 16108 raft_consensus.cc:3037] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:24:43.772635 16108 raft_consensus.cc:2454] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79752c65cf254ddab0e3e48e49d2beeb in term 1.
I20260924 15:24:43.772799 16184 leader_election.cc:288] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [CANDIDATE]: Term 1 election: Requested vote from peers 7e470510cfd74825a22c0f434559b021 (127.4.78.65:45117), c629cfca60004c6eae9ea999f758b932 (127.4.78.67:46835)
I20260924 15:24:43.772863 16184 raft_consensus.cc:2781] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:24:43.772907 16184 raft_consensus.cc:686] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 1 LEADER]: Becoming Leader. State: Replica: 79752c65cf254ddab0e3e48e49d2beeb, State: Running, Role: LEADER
I20260924 15:24:43.772992 16184 consensus_queue.cc:237] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } }
I20260924 15:24:43.773801 15668 catalog_manager.cc:5495] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 79752c65cf254ddab0e3e48e49d2beeb (127.4.78.66). New cstate: current_term: 1 leader_uuid: "79752c65cf254ddab0e3e48e49d2beeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "79752c65cf254ddab0e3e48e49d2beeb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 } health_report { overall_health: UNKNOWN } } }
I20260924 15:24:43.784300 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.781264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48822 (local address 127.4.78.66:44831)
0924 15:24:43.781299 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:24:43.781301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:24:43.781307 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:24:43.781337 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:24:43.781337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:24:43.781355 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:24:43.781410 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:24:43.781580 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.782280 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.783302 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:24:43.783586 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:24:43.783640 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:24:43.783813 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:24:43.783923 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:24:43.783986 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:24:43.784195 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:24:43.784215 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:24:43.786376 16108 raft_consensus.cc:1261] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 1 FOLLOWER]: Refusing update from remote peer 79752c65cf254ddab0e3e48e49d2beeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:24:43.786784 16184 consensus_queue.cc:1035] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c629cfca60004c6eae9ea999f758b932" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:24:43.788805 15836 raft_consensus.cc:1261] T 14c0650af57241b795c84505dd4e9665 P 7e470510cfd74825a22c0f434559b021 [term 1 FOLLOWER]: Refusing update from remote peer 79752c65cf254ddab0e3e48e49d2beeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:24:43.789129 16184 consensus_queue.cc:1035] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e470510cfd74825a22c0f434559b021" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 45117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:24:43.792855 16188 mvcc.cc:205] Tried to move back new op lower bound from 7332919229586124800 to 7332919229534699520. Current Snapshot: MvccSnapshot[applied={T|T < 7332919229586124800}]
I20260924 15:24:43.793426 16191 mvcc.cc:205] Tried to move back new op lower bound from 7332919229586124800 to 7332919229534699520. Current Snapshot: MvccSnapshot[applied={T|T < 7332919229586124800}]
I20260924 15:24:43.795548 16189 mvcc.cc:205] Tried to move back new op lower bound from 7332919229586124800 to 7332919229534699520. Current Snapshot: MvccSnapshot[applied={T|T < 7332919229586124800}]
I20260924 15:24:43.797864 15668 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:24:43.797946 15668 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:24:43.798691 15668 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9fa4f847d03b4c10aa79d7bba1e1642a: Sending DeleteTablet for 3 replicas of tablet 14c0650af57241b795c84505dd4e9665
I20260924 15:24:43.799088 15816 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c0650af57241b795c84505dd4e9665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:24:43 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:40868
I20260924 15:24:43.799299 16091 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c0650af57241b795c84505dd4e9665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:24:43 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:49518
I20260924 15:24:43.799672 15954 tablet_service.cc:1514] Processing DeleteTablet for tablet 14c0650af57241b795c84505dd4e9665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:24:43 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:48814
I20260924 15:24:43.800220 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 15750
I20260924 15:24:43.808202 16198 tablet_replica.cc:330] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: stopping tablet replica
I20260924 15:24:43.808357 16199 tablet_replica.cc:330] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: stopping tablet replica
I20260924 15:24:43.808532 16199 raft_consensus.cc:2229] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:24:43.808678 16198 raft_consensus.cc:2229] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:24:43.808852 16199 raft_consensus.cc:2258] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:24:43.808965 16198 raft_consensus.cc:2258] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:24:43.809600 16199 ts_tablet_manager.cc:1905] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:24:43.813148 16199 ts_tablet_manager.cc:1918] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:24:43.813390 16199 log.cc:1196] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/14c0650af57241b795c84505dd4e9665
I20260924 15:24:43.813766 16198 ts_tablet_manager.cc:1905] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:24:43.815377 16198 ts_tablet_manager.cc:1918] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:24:43.815503 16198 log.cc:1196] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/14c0650af57241b795c84505dd4e9665
I20260924 15:24:43.816107 16198 ts_tablet_manager.cc:1939] T 14c0650af57241b795c84505dd4e9665 P c629cfca60004c6eae9ea999f758b932: Deleting consensus metadata
I20260924 15:24:43.818089 16199 ts_tablet_manager.cc:1939] T 14c0650af57241b795c84505dd4e9665 P 79752c65cf254ddab0e3e48e49d2beeb: Deleting consensus metadata
I20260924 15:24:43.818542 15653 catalog_manager.cc:4841] TS 79752c65cf254ddab0e3e48e49d2beeb (127.4.78.66:44831): tablet 14c0650af57241b795c84505dd4e9665 (table test-table [id=d6eea320849a48faaed59d49c2b845d1]) successfully deleted
I20260924 15:24:43.819051 15653 catalog_manager.cc:4841] TS c629cfca60004c6eae9ea999f758b932 (127.4.78.67:46835): tablet 14c0650af57241b795c84505dd4e9665 (table test-table [id=d6eea320849a48faaed59d49c2b845d1]) successfully deleted
W20260924 15:24:43.819397 15653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:45117 (error 108)
I20260924 15:24:43.826107 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 15888
I20260924 15:24:43.827320 16170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:24:43.827188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:45117 (local address 127.0.0.1:40878)
0924 15:24:43.827241 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:24:43.827282 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:24:43.827523 15653 catalog_manager.cc:4584] TS 7e470510cfd74825a22c0f434559b021 (127.4.78.65:45117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14c0650af57241b795c84505dd4e9665: Network error: Client connection negotiation failed: client connection to 127.4.78.65:45117: connect: Connection refused (error 111)
I20260924 15:24:43.834235 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 16027
I20260924 15:24:43.841873 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 15637
I20260924 15:24:43.846206 4409 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 15:24:44.190214 4409 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 15:24:44.190321 4409 mini_ranger.cc:66] Stopping Ranger...
I20260924 15:24:44.534795 4409 mini_ranger.cc:68] Stopped Ranger
2026-09-24 15:24:44.535 UTC [15218] LOG: received smart shutdown request
2026-09-24 15:24:44.539 UTC [15218] LOG: background worker "logical replication launcher" (PID 15227) exited with exit code 1
2026-09-24 15:24:44.539 UTC [15222] LOG: shutting down
2026-09-24 15:24:44.702 UTC [15218] LOG: database system is shut down
2026-09-24T15:24:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46811 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:24:44 dist-test-slave-l766 krb5kdc[16209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:24:44 dist-test-slave-l766 krb5kdc[16209](info): set up 2 sockets
Sep 24 15:24:44 dist-test-slave-l766 krb5kdc[16209](info): commencing operation
krb5kdc: starting...
W20260924 15:24:45.860699 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:24:45 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:24:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:24:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.78.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.78.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/rangeradmin_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/rangeradmin_127.4.78.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.78.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.78.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/rangerlookup_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/rangerlookup_127.4.78.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.78.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.78.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/HTTP_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/HTTP_127.4.78.84@KRBTEST.COM.keytab.
I20260924 15:24:46.021621 4409 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskwmKTEk/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/postgres -l logfile start
2026-09-24 15:24:48.542 UTC [16240] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 15:24:48.543 UTC [16240] LOG: listening on IPv4 address "127.4.78.84", port 43659
2026-09-24 15:24:48.546 UTC [16240] LOG: listening on Unix socket "/tmp/.s.PGSQL.43659"
2026-09-24 15:24:48.567 UTC [16243] LOG: database system was shut down at 2026-09-24 15:24:47 UTC
2026-09-24 15:24:48.571 UTC [16240] LOG: database system is ready to accept connections
I20260924 15:24:49.540729 4409 mini_postgres.cc:96] Postgres bound to 43659
2026-09-24 15:24:49.546 UTC [16251] FATAL: database "slave" does not exist
127.4.78.84:43659 - accepting connections
I20260924 15:24:49.547412 4409 mini_ranger.cc:161] Starting Ranger...
I20260924 15:24:49.561775 4409 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 15:24:50.035458 4409 mini_ranger.cc:87] Created ranger Postgres database
I20260924 15:24:50.035543 4409 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-admin
2026-09-24 15:24:50,301 [I] DB FLAVOR :POSTGRES
2026-09-24 15:24:50,301 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:24:50,302 [I] Checking connection..
2026-09-24 15:24:50,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 15:24:50,554 [I] Checking connection passed.
2026-09-24 15:24:50,555 [I] --------- Verifying version history table ---------
2026-09-24 15:24:50,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:24:50,803 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:24:50,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:24:51,037 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:24:51,037 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 15:24:51,038 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 15:24:51.266 UTC [16315] WARNING: there is no transaction in progress
2026-09-24 15:24:51,272 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 15:24:51,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:24:51,530 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 15:24:51,530 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:24:51,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 15:24:51,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 15:24:52,029 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-l766.c.gcp-upstream.internal','N') ;"
2026-09-24 15:24:52,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:24:52,514 [I] Table x_portal_user does not exist in database ranger
2026-09-24 15:24:52,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:24:52,773 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 15:24:52,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:24:53,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:24:53,272 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 15:24:53,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 15:24:54.196 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.211 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.227 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.242 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.257 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.274 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.326 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.332 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.346 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.355 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.364 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.377 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.386 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.392 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.399 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:54.405 UTC [16424] WARNING: there is no transaction in progress
2026-09-24 15:24:55,026 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 15:24:55,026 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 15:24:55,274 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 15:24:55,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:24:55,524 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 15:24:55,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:24:55,778 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 15:24:55,779 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:24:56,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:24:56,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-l766.c.gcp-upstream.internal';"
2026-09-24 15:24:56,494 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 15:24:56,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 15:24:56,720 [I] DB_PATCHES have already been applied
I20260924 15:24:56.726496 4409 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-admin:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskwmKTEk/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 15:24:56.726572 4409 mini_ranger.cc:193] Using host: 127.4.78.84
I20260924 15:24:56.729521 4409 mini_ranger.cc:227] Ranger admin URL: http://127.4.78.84:58841
Sep 24, 2026 3:24:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:24:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:24:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:24:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:24:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:24:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:24:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 15:24:57.842229 4409 mini_ranger.cc:160] Time spent starting Ranger: real 8.295s user 0.006s sys 0.002s
Sep 24, 2026 3:25:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:25:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 3:25:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:25:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:25:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 15:25:19.878739 4409 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.78.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.78.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/rangerkms_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.78.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/rangerkms_127.4.78.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.78.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.78.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.78.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/HTTP_127.4.78.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.78.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/HTTP_127.4.78.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 15:25:19 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263519, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 15:25:19.979478 4409 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 15:25:19.989661 4409 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 15:25:20.861157 4409 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 15:25:20.861249 4409 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms
I20260924 15:25:20.861268 4409 mini_ranger_kms.cc:227] Using postgres at 127.4.78.84:43659
2026-09-24 15:25:21,031 [I] DB FLAVOR :POSTGRES
2026-09-24 15:25:21,031 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:25:21,031 [I] Checking connection
2026-09-24 15:25:21,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 15:25:21,289 [I] connection success
2026-09-24 15:25:21,290 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 15:25:21,290 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 15:25:21,290 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 15:25:21,529 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 15:25:21,529 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:25:21,530 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 15:25:21,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.78.84:43659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 15:25:21,788 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 15:25:22.275000 4409 mini_ranger_kms.cc:310] Created kms service
I20260924 15:25:22.481515 4409 mini_ranger_kms.cc:326] Created kudu user
I20260924 15:25:22.526075 4409 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 15:25:22 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263519, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:22.899802 4409 mini_ranger_kms.cc:384] Added ranger policy
I20260924 15:25:22.899917 4409 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms:/tmp/dist-test-taskwmKTEk/build/release/bin/postgresql.jar:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskwmKTEk/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 15:25:22.899950 4409 mini_ranger_kms.cc:240] Using host: 127.4.78.84
I20260924 15:25:22.903409 4409 mini_ranger_kms.cc:276] Ranger KMS PID: 16605
I20260924 15:25:22.903492 4409 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.78.84:48225
Sep 24, 2026 3:25:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:25:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:25:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:25:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:25:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:25:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 15:25:25 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263525, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.78.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:25.191146 4409 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.212s user 0.000s sys 0.153s
I20260924 15:25:25.191263 4409 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 15:25:25.191316 4409 mini_ranger_kms.cc:401] 127.4.78.84:48225/kms/v1/keys
Sep 24 15:25:25 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263525, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.78.84@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
Sep 24 15:25:25 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263525, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.78.84@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
Sep 24 15:25:28 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:25:28.352717 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39763
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39763
--ranger_config_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:48225/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:28.448781 16656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:28.448974 16656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:28.448995 16656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:28.450191 16656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:28.450242 16656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:28.450256 16656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:28.450269 16656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:28.450281 16656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:28.451779 16656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.78.84:48225/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39763
--ranger_config_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:28.452011 16656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:28.452224 16656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:28.457046 16662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:28.457011 16661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:28.457146 16656 server_base.cc:1034] running on GCE node
W20260924 15:25:28.456977 16664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:28.457592 16656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:28.457942 16656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:28.459098 16656 hybrid_clock.cc:648] HybridClock initialized: now 1790263528459085 us; error 49 us; skew 500 ppm
Sep 24 15:25:28 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:28.468353 16656 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:25:28.468771 16656 webserver.cc:460] Webserver started at http://127.4.78.126:43047/ using document root <none> and password file <none>
I20260924 15:25:28.468987 16656 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:28.469045 16656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:28.469236 16656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:25:28 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263528, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:28.519236 16656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "94f936d6951249e2871a866cb62107f3"
format_stamp: "Formatted at 2026-09-24 15:25:28 on dist-test-slave-l766"
server_key: "f10b5f3a07b1281aa3c8b199352eee97"
server_key_iv: "2438819ba025ccc9f00795edc3667d8b"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:28.519730 16656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "94f936d6951249e2871a866cb62107f3"
format_stamp: "Formatted at 2026-09-24 15:25:28 on dist-test-slave-l766"
server_key: "f10b5f3a07b1281aa3c8b199352eee97"
server_key_iv: "2438819ba025ccc9f00795edc3667d8b"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:28.521584 16656 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260924 15:25:28.536949 16671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:28.537201 16656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:25:28.537314 16656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "94f936d6951249e2871a866cb62107f3"
format_stamp: "Formatted at 2026-09-24 15:25:28 on dist-test-slave-l766"
server_key: "f10b5f3a07b1281aa3c8b199352eee97"
server_key_iv: "2438819ba025ccc9f00795edc3667d8b"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:28.537412 16656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:28.556181 16656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:28.558387 16656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:28.558532 16656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:28.562961 16656 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:39763
I20260924 15:25:28.563006 16723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:39763 every 8 connection(s)
I20260924 15:25:28.563360 16656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:25:28.563974 16724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:28.566313 16724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Bootstrap starting.
I20260924 15:25:28.566905 16724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:28.567209 16724 log.cc:824] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:28.567876 16724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: No bootstrap required, opened a new log
I20260924 15:25:28.568883 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 16656
I20260924 15:25:28.568996 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:25:28.569195 4409 external_mini_cluster.cc:1428] Setting key db2175102d9b023089e29bb31f04c4bd
I20260924 15:25:28.570621 16724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } }
I20260924 15:25:28.570863 16724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:28.570941 16724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f936d6951249e2871a866cb62107f3, State: Initialized, Role: FOLLOWER
I20260924 15:25:28.571080 16724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } }
I20260924 15:25:28.571197 16724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:25:28.571254 16724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:25:28.571316 16724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:28.571967 16724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } }
I20260924 15:25:28.572072 16724 leader_election.cc:302] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94f936d6951249e2871a866cb62107f3; no voters:
I20260924 15:25:28.572237 16724 leader_election.cc:288] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:25:28.572358 16729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:28.572474 16724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:28.572542 16729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 1 LEADER]: Becoming Leader. State: Replica: 94f936d6951249e2871a866cb62107f3, State: Running, Role: LEADER
I20260924 15:25:28.572718 16729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } }
I20260924 15:25:28.573153 16730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94f936d6951249e2871a866cb62107f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } } }
I20260924 15:25:28.573258 16730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:28.573412 16731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f936d6951249e2871a866cb62107f3. Latest consensus state: current_term: 1 leader_uuid: "94f936d6951249e2871a866cb62107f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } } }
I20260924 15:25:28.573474 16731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:28.573460 16738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:25:28.573915 16724 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:25:28.577884 16738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 15:25:28 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:28.580716 16738 catalog_manager.cc:1347] Generated new cluster ID: 4fce72c2cc8f49e59d6b17f410a4b960
I20260924 15:25:28.580772 16738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:25:28.584830 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:28.569792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60362 (local address 127.4.78.126:39763)
0924 15:25:28.570019 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:28.570030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:28.570066 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:25:28.570586 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:28.570594 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:28.570658 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:28.570821 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:28.571684 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:28.573196 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:28.575659 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:28.577634 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:28.577878 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:28.581061 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:28.581065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:28.581065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:28.581075 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:28.583018 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:28.583384 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:28.583386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:28.583387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:28.583484 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:28.583650 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:28.583650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:28.583651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:28.583930 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:28.584429 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:28.584494 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:25:28.584610 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260924 15:25:28.607270 16738 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:25:28.608139 16738 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:25:28.616127 16738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Generated new TSK 0
I20260924 15:25:28.616329 16738 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:25:29.306046 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:48225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39763
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:29.412461 16769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:29.412802 16769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:29.412879 16769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:29.414580 16769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:29.414647 16769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:29.414693 16769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:25:29.416644 16769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.78.84:48225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:29.416879 16769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:29.417161 16769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:29.423228 16776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:29.423305 16778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:29.423230 16775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:29.424571 16769 server_base.cc:1034] running on GCE node
I20260924 15:25:29.424836 16769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:29.425135 16769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:29.426348 16769 hybrid_clock.cc:648] HybridClock initialized: now 1790263529426312 us; error 47 us; skew 500 ppm
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:29.429410 16769 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:25:29.429883 16769 webserver.cc:460] Webserver started at http://127.4.78.65:38887/ using document root <none> and password file <none>
I20260924 15:25:29.430126 16769 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:29.430171 16769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:29.430316 16769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:29.442153 16769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "4b6abe5cf5af48eb9deee990f92933ff"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "996eb1d57e71f949ebb157b4b6cebc63"
server_key_iv: "856989b3b12f2cc18e35cb6d0d11e76b"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.442724 16769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "4b6abe5cf5af48eb9deee990f92933ff"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "996eb1d57e71f949ebb157b4b6cebc63"
server_key_iv: "856989b3b12f2cc18e35cb6d0d11e76b"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.444793 16769 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:25:29.465921 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.468981 16769 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.001s
I20260924 15:25:29.469153 16769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "4b6abe5cf5af48eb9deee990f92933ff"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "996eb1d57e71f949ebb157b4b6cebc63"
server_key_iv: "856989b3b12f2cc18e35cb6d0d11e76b"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.469278 16769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:29.497675 16769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:29.500188 16769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:29.500337 16769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:29.500612 16769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:29.500980 16769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:29.501031 16769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.501070 16769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:29.501094 16769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.508459 16769 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:46627
I20260924 15:25:29.508905 16769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:25:29.510614 16899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:46627 every 8 connection(s)
I20260924 15:25:29.513717 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 16769
I20260924 15:25:29.513844 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:25:29.514031 4409 external_mini_cluster.cc:1428] Setting key b3449bff545bd363c19b7d9e9ce49649
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:29.522675 16902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:29.511913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39763 (local address 127.4.78.65:43935)
0924 15:25:29.512126 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:29.512169 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:29.512775 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:29.513237 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:29.513242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:29.513479 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:29.513781 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:29.513789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.514914 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.514915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:29.516186 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:29.516189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.516710 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.516711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:29.516847 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:29.517433 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:29.517444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:29.518992 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:29.521014 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:29.521021 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:29.521023 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:29.521332 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:29.521557 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:29.521558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:29.521558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:29.521683 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:29.521987 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:29.521994 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:29.522170 (+ 176us) client_negotiation.cc:769] Sending connection context
0924 15:25:29.522280 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:25:29.522421 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":94,"threads_started":1}
I20260924 15:25:29.523029 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:29.511928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:43935 (local address 127.4.78.126:39763)
0924 15:25:29.512963 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:29.512966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:29.512975 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:25:29.513003 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:29.513004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:29.513016 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:29.513089 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:29.513922 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.514812 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.516286 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.516561 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.516626 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:29.519109 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:29.519111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:29.519111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:29.519118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:29.520917 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:29.521423 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:29.521424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:29.521425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:29.521489 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:29.521753 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:29.521754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:29.521754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:29.521902 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:29.521961 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:29.522830 (+ 869us) server_negotiation.cc:299] Negotiation successful
0924 15:25:29.522917 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004,"thread_start_us":98,"threads_started":1}
I20260924 15:25:29.523422 16900 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39763
I20260924 15:25:29.523523 16900 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:29.523715 16900 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:29.524281 16688 ts_manager.cc:194] Registered new tserver with Master: 4b6abe5cf5af48eb9deee990f92933ff (127.4.78.65:46627)
I20260924 15:25:29.525156 16688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:43935
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:25:29.552985 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:48225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39763
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 15:25:29.648841 16908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:29.649091 16908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:29.649194 16908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:29.650712 16908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:29.650771 16908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:29.650808 16908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:25:29.652325 16908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.78.84:48225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:29.652575 16908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:29.652796 16908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:29.657601 16916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:29.657766 16914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:29.657613 16913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:29.658300 16908 server_base.cc:1034] running on GCE node
I20260924 15:25:29.658522 16908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:29.658776 16908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:29.659935 16908 hybrid_clock.cc:648] HybridClock initialized: now 1790263529659928 us; error 29 us; skew 500 ppm
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:29.662564 16908 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:25:29.663028 16908 webserver.cc:460] Webserver started at http://127.4.78.66:42437/ using document root <none> and password file <none>
I20260924 15:25:29.663266 16908 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:29.663333 16908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:29.663468 16908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:29.679797 16908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "09ab9689b9124864829aa2c5d5a5db07"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "90a9f6681b8072478680760a58af114a"
server_key_iv: "ffd764081db8e32f0c664d9702f4e2fc"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.680222 16908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "09ab9689b9124864829aa2c5d5a5db07"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "90a9f6681b8072478680760a58af114a"
server_key_iv: "ffd764081db8e32f0c664d9702f4e2fc"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.682600 16908 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 15:25:29.699050 16924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.699324 16908 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:25:29.699414 16908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "09ab9689b9124864829aa2c5d5a5db07"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "90a9f6681b8072478680760a58af114a"
server_key_iv: "ffd764081db8e32f0c664d9702f4e2fc"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.699512 16908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:29.714273 16908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:29.716346 16908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:29.716502 16908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:29.716789 16908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:29.717159 16908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:29.717209 16908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.717247 16908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:29.717281 16908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.726148 16908 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:43541
I20260924 15:25:29.726398 17041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:43541 every 8 connection(s)
I20260924 15:25:29.726675 16908 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:25:29.731359 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 16908
I20260924 15:25:29.731472 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:25:29.731640 4409 external_mini_cluster.cc:1428] Setting key ba83dc4231aa586dacaa5c2072853b60
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:29.737046 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:29.727490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:51545 (local address 127.4.78.126:39763)
0924 15:25:29.727541 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:29.727543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:29.727802 (+ 259us) server_negotiation.cc:407] Connection header received
0924 15:25:29.728456 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:29.728458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:29.728469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:29.728520 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:29.729263 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.729757 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.731269 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.731598 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.731694 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:29.734197 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:29.734199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:29.734199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:29.734209 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:29.735639 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:29.736064 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:29.736065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:29.736066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:29.736125 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:29.736377 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:29.736377 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:29.736378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:29.736494 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:29.736575 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:29.736862 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 15:25:29.736957 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:25:29.737223 17044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:29.727472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39763 (local address 127.4.78.66:51545)
0924 15:25:29.727637 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:29.727687 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:29.728339 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:29.728610 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:29.728614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:29.728862 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:29.729167 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:29.729174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.729872 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.729873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:29.731171 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:29.731175 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.731717 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.731718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:29.731854 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:29.732418 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:29.732428 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:29.734087 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:29.735741 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:29.735747 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:29.735750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:29.735972 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:29.736193 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:29.736194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:29.736194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:29.736307 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:29.736569 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:29.736575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:29.736728 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 15:25:29.736841 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 15:25:29.736980 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260924 15:25:29.737716 17042 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39763
I20260924 15:25:29.737847 17042 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:29.738031 17042 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:29.738435 16688 ts_manager.cc:194] Registered new tserver with Master: 09ab9689b9124864829aa2c5d5a5db07 (127.4.78.66:43541)
I20260924 15:25:29.738814 16688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:51545
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:25:29.761104 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:48225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39763
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:29.874187 17047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:29.874397 17047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:29.874430 17047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:29.875519 17047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:29.875581 17047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:29.875628 17047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:25:29.877033 17047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.78.84:48225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:29.877266 17047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:29.877482 17047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:29.882371 17055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:29.882359 17052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:29.882577 17047 server_base.cc:1034] running on GCE node
W20260924 15:25:29.882625 17053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:29.882831 17047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:29.883098 17047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:29.884276 17047 hybrid_clock.cc:648] HybridClock initialized: now 1790263529884254 us; error 34 us; skew 500 ppm
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:29.887177 17047 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:25:29.887817 17047 webserver.cc:460] Webserver started at http://127.4.78.67:42817/ using document root <none> and password file <none>
I20260924 15:25:29.888118 17047 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:29.888195 17047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:29.888348 17047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:29.907398 17047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "478c3262c8dc4ad3b0696f7b2a6880d0"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "eb17a83fbdc0f68a33b55308b77248f3"
server_key_iv: "574c1531e12f380f3500638b073f7a0a"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.907903 17047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "478c3262c8dc4ad3b0696f7b2a6880d0"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "eb17a83fbdc0f68a33b55308b77248f3"
server_key_iv: "574c1531e12f380f3500638b073f7a0a"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.910211 17047 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:25:29.923880 17062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.924057 17047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260924 15:25:29.924146 17047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "478c3262c8dc4ad3b0696f7b2a6880d0"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "eb17a83fbdc0f68a33b55308b77248f3"
server_key_iv: "574c1531e12f380f3500638b073f7a0a"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:29.924237 17047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:29.976424 17047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:29.978886 17047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:29.979050 17047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:29.979276 17047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:29.979660 17047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:29.979709 17047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.979741 17047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:29.979766 17047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:29.986742 17047 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:35003
I20260924 15:25:29.987190 17047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:25:29.987493 17176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:35003 every 8 connection(s)
Sep 24 15:25:29 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263529, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:29.996302 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 17047
I20260924 15:25:29.996416 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:25:29.996598 4409 external_mini_cluster.cc:1428] Setting key c13d821597eadca0199f79229d5862d9
I20260924 15:25:29.999377 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:29.988735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:38077 (local address 127.4.78.126:39763)
0924 15:25:29.988822 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:29.988825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:29.989008 (+ 183us) server_negotiation.cc:407] Connection header received
0924 15:25:29.989623 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:29.989624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:29.989637 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:29.989703 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:29.990527 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.991297 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.992818 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.993097 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.993147 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:29.995711 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:29.995712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:29.995712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:29.995720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:29.997663 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:29.998160 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:29.998161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:29.998161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:29.998224 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:29.998555 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:29.998557 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:29.998557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:29.998720 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:29.998798 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:29.999195 (+ 397us) server_negotiation.cc:299] Negotiation successful
0924 15:25:29.999282 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 15:25:29.999763 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:29.988203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39763 (local address 127.4.78.67:38077)
0924 15:25:29.988840 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:29.988888 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:29.989509 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:29.989879 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:29.989885 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:29.990124 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:29.990408 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:29.990416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.991420 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.991421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:29.992697 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:29.992700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:29.993257 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:29.993258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:29.993396 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:29.993971 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:29.993981 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:29.995594 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:29.997738 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:29.997745 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:29.997747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:29.998062 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:29.998301 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:29.998302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:29.998302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:29.998434 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:29.998821 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:29.998828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:29.999015 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 15:25:29.999128 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 15:25:29.999279 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":93,"threads_started":1}
I20260924 15:25:30.000370 17177 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39763
I20260924 15:25:30.000459 17177 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.000641 17177 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:30.001049 16688 ts_manager.cc:194] Registered new tserver with Master: 478c3262c8dc4ad3b0696f7b2a6880d0 (127.4.78.67:35003)
I20260924 15:25:30.001511 16688 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:38077
I20260924 15:25:30.008419 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:25:30.009302 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 16769
I20260924 15:25:30.017372 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 16908
I20260924 15:25:30.022571 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 17047
I20260924 15:25:30.028218 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 16656
I20260924 15:25:30.035066 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39763
--webserver_interface=127.4.78.126
--webserver_port=43047
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39763
--ranger_config_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:48225/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.78.84:48225/kms with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:30.115235 17180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:30.115428 17180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:30.115450 17180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:30.116532 17180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:30.116565 17180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:30.116580 17180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:30.116591 17180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:30.116603 17180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:30.118109 17180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.78.84:48225/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39763
--ranger_config_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=43047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:30.118285 17180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:30.118436 17180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:30.123214 17188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:30.123243 17185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:30.123261 17180 server_base.cc:1034] running on GCE node
W20260924 15:25:30.123212 17186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:30.123870 17180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:30.124150 17180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:30.125348 17180 hybrid_clock.cc:648] HybridClock initialized: now 1790263530125333 us; error 30 us; skew 500 ppm
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:30.128355 17180 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:25:30.128844 17180 webserver.cc:460] Webserver started at http://127.4.78.126:43047/ using document root <none> and password file <none>
I20260924 15:25:30.129067 17180 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:30.129146 17180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:30.175812 17180 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:30.176609 17196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.176807 17180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:30.176906 17180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "94f936d6951249e2871a866cb62107f3"
format_stamp: "Formatted at 2026-09-24 15:25:28 on dist-test-slave-l766"
server_key: "f10b5f3a07b1281aa3c8b199352eee97"
server_key_iv: "2438819ba025ccc9f00795edc3667d8b"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:30.177290 17180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:30.203765 17180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:30.205758 17180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:30.205957 17180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:30.210386 17180 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:39763
I20260924 15:25:30.210436 17248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:39763 every 8 connection(s)
I20260924 15:25:30.210822 17180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:25:30.212080 17249 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:25:30.212678 17249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Bootstrap starting.
I20260924 15:25:30.220423 17249 log.cc:824] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:30.220607 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 17180
I20260924 15:25:30.220865 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:48225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:46627
--local_ip_for_outbound_sockets=127.4.78.65
--tserver_master_addrs=127.4.78.126:39763
--webserver_port=38887
--webserver_interface=127.4.78.65
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.78.84:48225/kms with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:25:30.222664 17249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:25:30.222941 17249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Bootstrap complete.
I20260924 15:25:30.225406 17249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } }
I20260924 15:25:30.225634 17249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f936d6951249e2871a866cb62107f3, State: Initialized, Role: FOLLOWER
I20260924 15:25:30.225793 17249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } }
I20260924 15:25:30.225893 17249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:25:30.225970 17249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:25:30.226017 17249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:25:30.226987 17249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } }
I20260924 15:25:30.227068 17249 leader_election.cc:302] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94f936d6951249e2871a866cb62107f3; no voters:
I20260924 15:25:30.227252 17249 leader_election.cc:288] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:25:30.227286 17254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:25:30.227385 17254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [term 2 LEADER]: Becoming Leader. State: Replica: 94f936d6951249e2871a866cb62107f3, State: Running, Role: LEADER
I20260924 15:25:30.227499 17249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:30.227509 17254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } }
I20260924 15:25:30.227802 17254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f936d6951249e2871a866cb62107f3. Latest consensus state: current_term: 2 leader_uuid: "94f936d6951249e2871a866cb62107f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } } }
I20260924 15:25:30.227890 17254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:30.228034 17255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "94f936d6951249e2871a866cb62107f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f936d6951249e2871a866cb62107f3" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39763 } } }
I20260924 15:25:30.228094 17255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:30.228111 17258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:25:30.228650 17258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:25:30.228812 17258 catalog_manager.cc:1259] Loaded cluster ID: 4fce72c2cc8f49e59d6b17f410a4b960
I20260924 15:25:30.228859 17258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:25:30.229739 17249 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:25:30.230291 17258 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:25:30.232921 17258 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Loaded TSK: 0
I20260924 15:25:30.233515 17258 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:25:30.316876 17253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:30.317099 17253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:30.317139 17253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:30.318408 17253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:30.318472 17253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:30.318516 17253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:25:30.320065 17253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.78.84:48225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:46627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=38887
--tserver_master_addrs=127.4.78.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:30.320307 17253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:30.320544 17253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:30.325529 17288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:30.325515 17290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:30.325915 17287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:30.325978 17253 server_base.cc:1034] running on GCE node
I20260924 15:25:30.326236 17253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:30.326472 17253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:30.327631 17253 hybrid_clock.cc:648] HybridClock initialized: now 1790263530327624 us; error 28 us; skew 500 ppm
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:30.330410 17253 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:25:30.330850 17253 webserver.cc:460] Webserver started at http://127.4.78.65:38887/ using document root <none> and password file <none>
I20260924 15:25:30.331084 17253 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:30.331167 17253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:30.366267 17253 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.001s sys 0.000s
I20260924 15:25:30.376482 17298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.376710 17253 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260924 15:25:30.376808 17253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "4b6abe5cf5af48eb9deee990f92933ff"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "996eb1d57e71f949ebb157b4b6cebc63"
server_key_iv: "856989b3b12f2cc18e35cb6d0d11e76b"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:30.377245 17253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:30.394771 17253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:30.397203 17253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:30.397373 17253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:30.397647 17253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:30.398090 17253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:30.398145 17253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.398185 17253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:30.398216 17253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.404433 17253 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:46627
I20260924 15:25:30.404654 17412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:46627 every 8 connection(s)
I20260924 15:25:30.404891 17253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:25:30.413401 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 17253
I20260924 15:25:30.413700 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:48225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:43541
--local_ip_for_outbound_sockets=127.4.78.66
--tserver_master_addrs=127.4.78.126:39763
--webserver_port=42437
--webserver_interface=127.4.78.66
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.78.84:48225/kms with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:30.423398 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:30.410614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39763 (local address 127.4.78.65:46175)
0924 15:25:30.411146 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:30.411188 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:30.411714 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:30.412163 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:30.412167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:30.412367 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:30.412566 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:30.412570 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.414109 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.414110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:30.415423 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:30.415427 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.416441 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.416442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:30.416576 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:30.417142 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:30.417152 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:30.418911 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:30.421330 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:30.421336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:30.421338 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:30.421659 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:30.421991 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:30.421992 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:30.421993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:30.422126 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:30.422649 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:30.422656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:30.422818 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 15:25:30.422937 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 15:25:30.423105 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":89,"threads_started":1}
I20260924 15:25:30.424052 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:30.410907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:46175 (local address 127.4.78.126:39763)
0924 15:25:30.411106 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:30.411116 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:30.411312 (+ 196us) server_negotiation.cc:407] Connection header received
0924 15:25:30.411939 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:30.411944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:30.411989 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:30.412122 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:30.412687 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.413995 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.415533 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.416209 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.416321 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:30.419038 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:30.419042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:30.419042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:30.419055 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:30.421231 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:30.421755 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:30.421759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:30.421760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:30.421909 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:30.422221 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:30.422222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:30.422223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:30.422531 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:30.422617 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:30.423689 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0924 15:25:30.423809 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":107,"threads_started":1}
I20260924 15:25:30.424758 17413 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39763
I20260924 15:25:30.424854 17413 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.425053 17413 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
W20260924 15:25:30.426026 17413 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39763 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:25:30.426098 17413 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.426169 17413 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:30.426611 17413 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.426689 17413 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
W20260924 15:25:30.427057 17413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:25:30.522382 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:30.522607 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:30.522646 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:30.523800 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:30.523867 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:30.523908 17417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:25:30.525745 17417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.78.84:48225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:43541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=42437
--tserver_master_addrs=127.4.78.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:30.526021 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:30.526273 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:30.533459 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:30.533594 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:30.533468 17422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:30.534098 17417 server_base.cc:1034] running on GCE node
I20260924 15:25:30.534405 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:30.534693 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:30.535853 17417 hybrid_clock.cc:648] HybridClock initialized: now 1790263530535824 us; error 45 us; skew 500 ppm
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:30.538658 17417 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:25:30.539187 17417 webserver.cc:460] Webserver started at http://127.4.78.66:42437/ using document root <none> and password file <none>
I20260924 15:25:30.539494 17417 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:30.539608 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:30.555650 17417 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.001s
I20260924 15:25:30.556538 17433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.556711 17417 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:25:30.556799 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "09ab9689b9124864829aa2c5d5a5db07"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "90a9f6681b8072478680760a58af114a"
server_key_iv: "ffd764081db8e32f0c664d9702f4e2fc"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:30.557261 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:30.585341 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:30.587379 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:30.587563 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:30.587838 17417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:30.588222 17417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:30.588282 17417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.588325 17417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:30.588352 17417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.595209 17417 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:43541
I20260924 15:25:30.595371 17547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:43541 every 8 connection(s)
I20260924 15:25:30.595708 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:25:30.600714 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 17417
I20260924 15:25:30.601012 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.78.84:48225/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:35003
--local_ip_for_outbound_sockets=127.4.78.67
--tserver_master_addrs=127.4.78.126:39763
--webserver_port=42817
--webserver_interface=127.4.78.67
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.78.84:48225/kms with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:30.607170 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:30.596861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:40435 (local address 127.4.78.126:39763)
0924 15:25:30.596942 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:30.596944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:30.597114 (+ 170us) server_negotiation.cc:407] Connection header received
0924 15:25:30.597685 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:30.597694 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:30.597710 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:30.597790 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:30.598515 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.599307 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.600397 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.600706 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.600790 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:30.603375 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:30.603377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:30.603377 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:30.603386 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:30.605100 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:30.605606 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:30.605607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:30.605608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:30.605672 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:30.606010 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:30.606012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:30.606012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:30.606158 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:30.606224 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:30.606922 (+ 698us) server_negotiation.cc:299] Negotiation successful
0924 15:25:30.607025 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 15:25:30.607918 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:30.596813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39763 (local address 127.4.78.66:40435)
0924 15:25:30.596967 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:30.597007 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:30.597553 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:30.597907 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:30.597910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:30.598154 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:30.598413 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:30.598418 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.599410 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.599411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:30.600308 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:30.600310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.600884 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.600885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:30.601043 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:30.601649 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:30.601659 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:30.603273 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:30.605189 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:30.605196 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:30.605198 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:30.605515 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:30.605741 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:30.605741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:30.605742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:30.605929 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:30.606252 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:30.606259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:30.606440 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 15:25:30.606552 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 15:25:30.607601 (+ 1049us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260924 15:25:30.608896 17548 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39763
I20260924 15:25:30.609006 17548 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.609225 17548 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
W20260924 15:25:30.609988 17548 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39763 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:25:30.610119 17548 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.610234 17548 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:30.610728 17548 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.610862 17548 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
W20260924 15:25:30.611256 17548 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:25:30.700430 17551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:30.700642 17551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:30.700680 17551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:30.701903 17551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:30.701960 17551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:30.702003 17551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:25:30.703461 17551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.78.84:48225/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:35003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=42817
--tserver_master_addrs=127.4.78.126:39763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:30.703707 17551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:30.703935 17551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:30.709167 17559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:30.709174 17557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:30.709915 17551 server_base.cc:1034] running on GCE node
W20260924 15:25:30.710098 17556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:30.710351 17551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:30.710731 17551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:30.711885 17551 hybrid_clock.cc:648] HybridClock initialized: now 1790263530711868 us; error 33 us; skew 500 ppm
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:30.714367 17551 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:25:30.714768 17551 webserver.cc:460] Webserver started at http://127.4.78.67:42817/ using document root <none> and password file <none>
I20260924 15:25:30.714988 17551 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:30.715067 17551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:30.740815 17551 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:25:30.741770 17568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.741989 17551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:30.742082 17551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "478c3262c8dc4ad3b0696f7b2a6880d0"
format_stamp: "Formatted at 2026-09-24 15:25:29 on dist-test-slave-l766"
server_key: "eb17a83fbdc0f68a33b55308b77248f3"
server_key_iv: "574c1531e12f380f3500638b073f7a0a"
server_key_version: "kuduclusterkey@0"
I20260924 15:25:30.742468 17551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:30.764637 17551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:30.766855 17551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:30.767019 17551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:30.767302 17551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:30.767675 17551 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:30.767732 17551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.767772 17551 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:30.767804 17551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:30.775097 17551 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:35003
I20260924 15:25:30.775194 17681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:35003 every 8 connection(s)
I20260924 15:25:30.775574 17551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:25:30.777906 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 17551
Sep 24 15:25:30 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263530, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:30.786118 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:30.776373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:43731 (local address 127.4.78.126:39763)
0924 15:25:30.776430 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:30.776436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:30.776631 (+ 195us) server_negotiation.cc:407] Connection header received
0924 15:25:30.777144 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:30.777146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:30.777159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:30.777234 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:30.777739 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.778529 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.780016 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.780333 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.780381 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:30.782724 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:30.782726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:30.782726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:30.782732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:30.784478 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:30.784990 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:30.784992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:30.784992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:30.785054 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:30.785341 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:30.785343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:30.785343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:30.785504 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:30.785574 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:30.785936 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 15:25:30.786027 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:25:30.786307 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:30.776358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:39763 (local address 127.4.78.67:43731)
0924 15:25:30.776516 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:30.776553 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:30.777045 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:30.777291 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:30.777295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:30.777455 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:30.777638 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:30.777642 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.778631 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.778633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:30.779925 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:30.779929 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.780424 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.780426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:30.780559 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:30.781126 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:30.781135 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:30.782620 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:30.784597 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:30.784604 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:30.784606 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:30.784903 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:30.785126 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:30.785126 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:30.785127 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:30.785260 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:30.785588 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:30.785594 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:30.785752 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 15:25:30.785896 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 15:25:30.786066 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260924 15:25:30.786743 17682 heartbeater.cc:347] Connected to a master server at 127.4.78.126:39763
I20260924 15:25:30.786842 17682 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.787025 17682 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
W20260924 15:25:30.787734 17682 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:39763 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:25:30.787808 17682 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.787873 17682 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:30.788256 17682 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:30.788328 17682 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
W20260924 15:25:30.788751 17682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:25:30.791075 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:30.778345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.4.78.126:39763)
0924 15:25:30.778617 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:30.778618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:30.778624 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:25:30.778641 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:30.778641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:30.778649 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:30.778727 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:30.780269 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.781509 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.782726 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:30.783030 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:30.783167 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:30.784571 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:30.784572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:30.784573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:30.784580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:30.787265 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:30.787698 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:30.787699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:30.787699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:30.787765 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:30.788051 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:30.788052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:30.788052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:30.788175 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:30.788311 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:30.790879 (+ 2568us) server_negotiation.cc:299] Negotiation successful
0924 15:25:30.790972 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":64,"threads_started":1}
Sep 24 15:25:31 dist-test-slave-l766 krb5kdc[16209](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263531, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:31.427243 17413 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:31.427407 17413 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:31.427826 17213 ts_manager.cc:194] Registered new tserver with Master: 4b6abe5cf5af48eb9deee990f92933ff (127.4.78.65:46627)
I20260924 15:25:31.428881 17213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:46175
I20260924 15:25:31.611493 17548 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:31.611668 17548 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:31.611979 17213 ts_manager.cc:194] Registered new tserver with Master: 09ab9689b9124864829aa2c5d5a5db07 (127.4.78.66:43541)
I20260924 15:25:31.612545 17213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:40435
Sep 24 15:25:31 dist-test-slave-l766 krb5kdc[16209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263531, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for HTTP/127.4.78.84@KRBTEST.COM
I20260924 15:25:31.788935 17682 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:31.789088 17682 heartbeater.cc:511] Master 127.4.78.126:39763 requested a full tablet report, sending...
I20260924 15:25:31.789419 17213 ts_manager.cc:194] Registered new tserver with Master: 478c3262c8dc4ad3b0696f7b2a6880d0 (127.4.78.67:35003)
I20260924 15:25:31.789965 17213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:43731
I20260924 15:25:31.791996 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:25:31.801867 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.792943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.4.78.126:39763)
0924 15:25:31.793062 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:31.793064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:31.793983 (+ 919us) server_negotiation.cc:407] Connection header received
0924 15:25:31.794058 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:31.794059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:31.794072 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:31.794135 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:31.794837 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.795577 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.796643 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.796962 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.797010 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.798623 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:31.798624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:31.798624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:31.798631 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:31.800400 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:31.800808 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:31.800809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:31.800810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:31.800868 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:31.801061 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:31.801062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:31.801063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:31.801182 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:31.801269 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:31.801648 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 15:25:31.801727 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":67,"threads_started":1}
I20260924 15:25:31.804296 17213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:25:31.806473 17213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:25:31.818704 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.809688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.4.78.66:43541)
0924 15:25:31.809864 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:31.809870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:31.814014 (+ 4144us) server_negotiation.cc:407] Connection header received
0924 15:25:31.814142 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:31.814145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:31.814161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:31.814220 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:31.814557 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.815665 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.817244 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.818199 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.818250 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.818307 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:31.818581 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 15:25:31.818622 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
I20260924 15:25:31.819195 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.809603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43541 (local address 127.0.0.1:57136)
0924 15:25:31.813930 (+ 4327us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:31.813941 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:31.814047 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:31.814345 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:31.814346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:31.814348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:31.814417 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.814419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.815782 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.815783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.817139 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.817142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.818376 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.818377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.818453 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.818466 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:25:31.819056 (+ 590us) client_negotiation.cc:240] Negotiation successful
0924 15:25:31.819121 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4308,"mutex_wait_us":47,"thread_start_us":66,"threads_started":1}
I20260924 15:25:31.820045 17481 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7856c34e2b49cc85e35eff5fc961ef (DEFAULT_TABLE table=test-table [id=dd8867ae0e3e492eaafacadcfd79d8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:31.820407 17481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7856c34e2b49cc85e35eff5fc961ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:31.821429 17710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.809393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:35003 (local address 127.0.0.1:37416)
0924 15:25:31.810207 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:31.810249 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:31.810309 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:31.814128 (+ 3819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:31.814131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:31.814134 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:31.814456 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.814459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.815786 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.815787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.817422 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.817424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.821193 (+ 3769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.821195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.821255 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.821259 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:25:31.821320 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:25:31.821357 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"mutex_wait_us":45,"thread_start_us":77,"threads_started":1}
I20260924 15:25:31.821882 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.809497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.4.78.67:35003)
0924 15:25:31.813943 (+ 4446us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:31.813947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:31.813959 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:25:31.813995 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:31.813997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:31.814014 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:31.814538 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:31.814640 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.815683 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.817620 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.821101 (+ 3481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.821175 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.821227 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:31.821702 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 15:25:31.821747 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4408,"thread_start_us":63,"threads_started":1}
I20260924 15:25:31.822652 17616 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7856c34e2b49cc85e35eff5fc961ef (DEFAULT_TABLE table=test-table [id=dd8867ae0e3e492eaafacadcfd79d8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:31.822994 17616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7856c34e2b49cc85e35eff5fc961ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:31.823608 17716 tablet_bootstrap.cc:492] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Bootstrap starting.
I20260924 15:25:31.824302 17716 tablet_bootstrap.cc:654] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:31.824679 17716 log.cc:824] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:31.826175 17718 tablet_bootstrap.cc:492] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: Bootstrap starting.
I20260924 15:25:31.826881 17718 tablet_bootstrap.cc:654] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:31.827256 17718 log.cc:824] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:31.827412 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.809849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.4.78.65:46627)
0924 15:25:31.809943 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:31.809947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:31.817583 (+ 7636us) server_negotiation.cc:407] Connection header received
0924 15:25:31.817953 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:31.817956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:31.817973 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:31.818158 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:31.818376 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.819412 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.821874 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.826974 (+ 5100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.827022 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.827070 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:31.827287 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 15:25:31.827332 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69,"thread_start_us":48,"threads_started":1}
I20260924 15:25:31.827771 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.809742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:46627 (local address 127.0.0.1:58598)
0924 15:25:31.817521 (+ 7779us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:31.817530 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:31.817851 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:31.818267 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:31.818268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:31.818270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:31.818306 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.818307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.819622 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.819623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.821774 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.821777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.827132 (+ 5355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.827133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.827196 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.827200 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:25:31.827648 (+ 448us) client_negotiation.cc:240] Negotiation successful
0924 15:25:31.827707 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7764,"thread_start_us":85,"threads_started":1}
I20260924 15:25:31.828459 17346 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7856c34e2b49cc85e35eff5fc961ef (DEFAULT_TABLE table=test-table [id=dd8867ae0e3e492eaafacadcfd79d8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:31.828778 17346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7856c34e2b49cc85e35eff5fc961ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:31.830581 17716 tablet_bootstrap.cc:492] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: No bootstrap required, opened a new log
I20260924 15:25:31.830718 17716 ts_tablet_manager.cc:1397] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 15:25:31.833676 17716 raft_consensus.cc:348] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.833951 17716 raft_consensus.cc:374] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:31.834033 17716 raft_consensus.cc:729] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ab9689b9124864829aa2c5d5a5db07, State: Initialized, Role: FOLLOWER
I20260924 15:25:31.834201 17716 consensus_queue.cc:260] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.834388 17718 tablet_bootstrap.cc:492] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: No bootstrap required, opened a new log
I20260924 15:25:31.834456 17718 ts_tablet_manager.cc:1397] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260924 15:25:31.834599 17716 ts_tablet_manager.cc:1428] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 15:25:31.834754 17548 heartbeater.cc:503] Master 127.4.78.126:39763 was elected leader, sending a full tablet report...
I20260924 15:25:31.836057 17720 tablet_bootstrap.cc:492] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff: Bootstrap starting.
I20260924 15:25:31.836799 17720 tablet_bootstrap.cc:654] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:31.836966 17718 raft_consensus.cc:348] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.837165 17720 log.cc:824] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:31.837180 17718 raft_consensus.cc:374] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:31.837229 17718 raft_consensus.cc:729] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 478c3262c8dc4ad3b0696f7b2a6880d0, State: Initialized, Role: FOLLOWER
I20260924 15:25:31.837369 17718 consensus_queue.cc:260] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.837667 17718 ts_tablet_manager.cc:1428] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:25:31.837843 17682 heartbeater.cc:503] Master 127.4.78.126:39763 was elected leader, sending a full tablet report...
I20260924 15:25:31.838248 17720 tablet_bootstrap.cc:492] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff: No bootstrap required, opened a new log
I20260924 15:25:31.838311 17720 ts_tablet_manager.cc:1397] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:25:31.840310 17720 raft_consensus.cc:348] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.840441 17720 raft_consensus.cc:374] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:31.840466 17720 raft_consensus.cc:729] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6abe5cf5af48eb9deee990f92933ff, State: Initialized, Role: FOLLOWER
I20260924 15:25:31.840564 17720 consensus_queue.cc:260] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.840778 17720 ts_tablet_manager.cc:1428] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:25:31.840878 17413 heartbeater.cc:503] Master 127.4.78.126:39763 was elected leader, sending a full tablet report...
W20260924 15:25:31.846738 17549 tablet.cc:2369] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:25:31.866930 17724 raft_consensus.cc:482] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:31.867043 17724 raft_consensus.cc:504] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.867409 17724 leader_election.cc:288] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 478c3262c8dc4ad3b0696f7b2a6880d0 (127.4.78.67:35003), 09ab9689b9124864829aa2c5d5a5db07 (127.4.78.66:43541)
I20260924 15:25:31.872640 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.867926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:41011 (local address 127.4.78.67:35003)
0924 15:25:31.867963 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:31.867965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:31.868073 (+ 108us) server_negotiation.cc:407] Connection header received
0924 15:25:31.868163 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:31.868164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:31.868174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:31.868234 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:31.868469 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.869176 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.870691 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.871294 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.871342 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.871353 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:31.872515 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0924 15:25:31.872559 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:25:31.872980 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.867599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:35003 (local address 127.4.78.65:41011)
0924 15:25:31.868023 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:31.868033 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:31.868096 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:31.868318 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:31.868320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:31.868321 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:31.868399 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.868400 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.869271 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.869272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.870557 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.870561 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.872351 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.872352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.872423 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.872427 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:25:31.872838 (+ 411us) client_negotiation.cc:240] Negotiation successful
0924 15:25:31.872884 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":107,"threads_started":1}
I20260924 15:25:31.873366 17630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7856c34e2b49cc85e35eff5fc961ef" candidate_uuid: "4b6abe5cf5af48eb9deee990f92933ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" is_pre_election: true
I20260924 15:25:31.873518 17630 raft_consensus.cc:2454] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b6abe5cf5af48eb9deee990f92933ff in term 0.
I20260924 15:25:31.873762 17302 leader_election.cc:302] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 478c3262c8dc4ad3b0696f7b2a6880d0, 4b6abe5cf5af48eb9deee990f92933ff; no voters:
I20260924 15:25:31.873782 17713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.867850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:35045 (local address 127.4.78.66:43541)
0924 15:25:31.867893 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:31.867895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:31.868479 (+ 584us) server_negotiation.cc:407] Connection header received
0924 15:25:31.868584 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:31.868585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:31.868598 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:31.868627 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:31.868836 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.869565 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.871874 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.872428 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.872479 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.872490 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:31.873679 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0924 15:25:31.873720 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:25:31.873900 17724 raft_consensus.cc:2781] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:25:31.873940 17724 raft_consensus.cc:482] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:25:31.873960 17724 raft_consensus.cc:3037] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:31.874094 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.867792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43541 (local address 127.4.78.65:35045)
0924 15:25:31.868408 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:31.868420 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:31.868508 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:31.868692 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:31.868692 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:31.868694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:31.868762 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.868764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.869894 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.869895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.871047 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:31.871050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.872739 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.872740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:31.873578 (+ 838us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.873583 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:25:31.873972 (+ 389us) client_negotiation.cc:240] Negotiation successful
0924 15:25:31.874014 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":52,"threads_started":1}
I20260924 15:25:31.874397 17501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7856c34e2b49cc85e35eff5fc961ef" candidate_uuid: "4b6abe5cf5af48eb9deee990f92933ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ab9689b9124864829aa2c5d5a5db07" is_pre_election: true
I20260924 15:25:31.874534 17501 raft_consensus.cc:2454] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b6abe5cf5af48eb9deee990f92933ff in term 0.
I20260924 15:25:31.874802 17724 raft_consensus.cc:504] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.874914 17724 leader_election.cc:288] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [CANDIDATE]: Term 1 election: Requested vote from peers 478c3262c8dc4ad3b0696f7b2a6880d0 (127.4.78.67:35003), 09ab9689b9124864829aa2c5d5a5db07 (127.4.78.66:43541)
I20260924 15:25:31.875151 17501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7856c34e2b49cc85e35eff5fc961ef" candidate_uuid: "4b6abe5cf5af48eb9deee990f92933ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ab9689b9124864829aa2c5d5a5db07"
I20260924 15:25:31.875216 17501 raft_consensus.cc:3037] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:31.875249 17630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7856c34e2b49cc85e35eff5fc961ef" candidate_uuid: "4b6abe5cf5af48eb9deee990f92933ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0"
I20260924 15:25:31.875349 17630 raft_consensus.cc:3037] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:31.876051 17501 raft_consensus.cc:2454] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b6abe5cf5af48eb9deee990f92933ff in term 1.
I20260924 15:25:31.876266 17302 leader_election.cc:302] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ab9689b9124864829aa2c5d5a5db07, 4b6abe5cf5af48eb9deee990f92933ff; no voters:
I20260924 15:25:31.876367 17724 raft_consensus.cc:2781] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:31.876385 17630 raft_consensus.cc:2454] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b6abe5cf5af48eb9deee990f92933ff in term 1.
I20260924 15:25:31.876505 17724 raft_consensus.cc:686] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 1 LEADER]: Becoming Leader. State: Replica: 4b6abe5cf5af48eb9deee990f92933ff, State: Running, Role: LEADER
I20260924 15:25:31.876580 17724 consensus_queue.cc:237] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } }
I20260924 15:25:31.877318 17212 catalog_manager.cc:5495] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b6abe5cf5af48eb9deee990f92933ff (127.4.78.65). New cstate: current_term: 1 leader_uuid: "4b6abe5cf5af48eb9deee990f92933ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b6abe5cf5af48eb9deee990f92933ff" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46627 } health_report { overall_health: HEALTHY } } }
I20260924 15:25:31.887346 17715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.884343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.4.78.65:46627)
0924 15:25:31.884384 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:31.884386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:31.884395 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:25:31.884427 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:31.884428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:31.884447 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:31.884529 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:25:31.884681 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.885351 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.886415 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:31.886683 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:31.886736 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:31.886907 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:25:31.887016 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:25:31.887078 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:31.887271 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 15:25:31.887287 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:25:31.889425 17501 raft_consensus.cc:1261] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 1 FOLLOWER]: Refusing update from remote peer 4b6abe5cf5af48eb9deee990f92933ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:31.889839 17727 consensus_queue.cc:1035] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ab9689b9124864829aa2c5d5a5db07" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:31.890404 17630 raft_consensus.cc:1261] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 1 FOLLOWER]: Refusing update from remote peer 4b6abe5cf5af48eb9deee990f92933ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:31.891901 17724 consensus_queue.cc:1035] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "478c3262c8dc4ad3b0696f7b2a6880d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:31.895905 17731 mvcc.cc:205] Tried to move back new op lower bound from 7332919426616090624 to 7332919426567053312. Current Snapshot: MvccSnapshot[applied={T|T < 7332919426616090624}]
I20260924 15:25:31.895898 17730 mvcc.cc:205] Tried to move back new op lower bound from 7332919426616090624 to 7332919426567053312. Current Snapshot: MvccSnapshot[applied={T|T < 7332919426616090624 or (T in {7332919426616090624})}]
I20260924 15:25:31.896740 17212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:25:31.896814 17733 mvcc.cc:205] Tried to move back new op lower bound from 7332919426616090624 to 7332919426567053312. Current Snapshot: MvccSnapshot[applied={T|T < 7332919426616090624}]
I20260924 15:25:31.896842 17212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:25:31.897604 17212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94f936d6951249e2871a866cb62107f3: Sending DeleteTablet for 3 replicas of tablet 9c7856c34e2b49cc85e35eff5fc961ef
I20260924 15:25:31.898007 17616 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7856c34e2b49cc85e35eff5fc961ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:31 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:37416
I20260924 15:25:31.898074 17481 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7856c34e2b49cc85e35eff5fc961ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:31 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:57136
I20260924 15:25:31.898227 17346 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7856c34e2b49cc85e35eff5fc961ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:31 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:58598
I20260924 15:25:31.898442 17742 tablet_replica.cc:330] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff: stopping tablet replica
I20260924 15:25:31.898525 17742 raft_consensus.cc:2229] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:25:31.898636 17742 raft_consensus.cc:2258] T 9c7856c34e2b49cc85e35eff5fc961ef P 4b6abe5cf5af48eb9deee990f92933ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:31.898804 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 17253
I20260924 15:25:31.900717 17740 tablet_replica.cc:330] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: stopping tablet replica
I20260924 15:25:31.900828 17740 raft_consensus.cc:2229] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:25:31.900955 17740 raft_consensus.cc:2258] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:31.901577 17740 ts_tablet_manager.cc:1905] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:31.903257 17741 tablet_replica.cc:330] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: stopping tablet replica
I20260924 15:25:31.903395 17741 raft_consensus.cc:2229] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:25:31.903537 17741 raft_consensus.cc:2258] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:31.903720 17740 ts_tablet_manager.cc:1918] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:31.903807 17740 log.cc:1196] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/9c7856c34e2b49cc85e35eff5fc961ef
I20260924 15:25:31.904103 17740 ts_tablet_manager.cc:1939] T 9c7856c34e2b49cc85e35eff5fc961ef P 478c3262c8dc4ad3b0696f7b2a6880d0: Deleting consensus metadata
I20260924 15:25:31.904626 17200 catalog_manager.cc:4841] TS 478c3262c8dc4ad3b0696f7b2a6880d0 (127.4.78.67:35003): tablet 9c7856c34e2b49cc85e35eff5fc961ef (table test-table [id=dd8867ae0e3e492eaafacadcfd79d8f0]) successfully deleted
I20260924 15:25:31.905057 17741 ts_tablet_manager.cc:1905] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:31.908043 17741 ts_tablet_manager.cc:1918] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:31.908145 17741 log.cc:1196] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/9c7856c34e2b49cc85e35eff5fc961ef
I20260924 15:25:31.908416 17741 ts_tablet_manager.cc:1939] T 9c7856c34e2b49cc85e35eff5fc961ef P 09ab9689b9124864829aa2c5d5a5db07: Deleting consensus metadata
I20260924 15:25:31.908939 17200 catalog_manager.cc:4841] TS 09ab9689b9124864829aa2c5d5a5db07 (127.4.78.66:43541): tablet 9c7856c34e2b49cc85e35eff5fc961ef (table test-table [id=dd8867ae0e3e492eaafacadcfd79d8f0]) successfully deleted
W20260924 15:25:31.909695 17200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:46627 (error 108)
I20260924 15:25:31.910460 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 17417
I20260924 15:25:31.918139 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 17551
I20260924 15:25:31.918102 17714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:31.917953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:46627 (local address 127.0.0.1:58628)
0924 15:25:31.918017 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:31.918065 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:46627: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 15:25:31.921178 17200 catalog_manager.cc:4584] TS 4b6abe5cf5af48eb9deee990f92933ff (127.4.78.65:46627): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c7856c34e2b49cc85e35eff5fc961ef: Network error: Client connection negotiation failed: client connection to 127.4.78.65:46627: connect: Connection refused (error 111)
I20260924 15:25:31.925649 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 17180
I20260924 15:25:31.933552 4409 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 15:25:32.268663 4409 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 15:25:32.268754 4409 mini_ranger.cc:66] Stopping Ranger...
I20260924 15:25:32.612672 4409 mini_ranger.cc:68] Stopped Ranger
2026-09-24 15:25:32.612 UTC [16240] LOG: received smart shutdown request
2026-09-24 15:25:32.617 UTC [16240] LOG: background worker "logical replication launcher" (PID 16249) exited with exit code 1
2026-09-24 15:25:32.617 UTC [16244] LOG: shutting down
2026-09-24 15:25:32.781 UTC [16240] LOG: database system is shut down
2026-09-24T15:25:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48077 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:25:32 dist-test-slave-l766 krb5kdc[17752](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:25:32 dist-test-slave-l766 krb5kdc[17752](info): set up 2 sockets
Sep 24 15:25:32 dist-test-slave-l766 krb5kdc[17752](info): commencing operation
krb5kdc: starting...
W20260924 15:25:33.936666 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:25:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:25:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:25:34.056905 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:34259
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:34259,127.4.78.125:42917,127.4.78.124:37719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:34.137207 17765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:34.137406 17765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:34.137427 17765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:34.138512 17765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:34.138545 17765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:34.138559 17765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:34.138572 17765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:34.138584 17765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:34.139955 17765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:34259,127.4.78.125:42917,127.4.78.124:37719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:34.140136 17765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:34.140303 17765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:34.145064 17774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:34.145064 17771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:34.145117 17772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:34.145356 17765 server_base.cc:1034] running on GCE node
I20260924 15:25:34.145562 17765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:34.145884 17765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:34.147035 17765 hybrid_clock.cc:648] HybridClock initialized: now 1790263534147021 us; error 30 us; skew 500 ppm
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:34.152884 17765 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:25:34.153342 17765 webserver.cc:460] Webserver started at http://127.4.78.126:40445/ using document root <none> and password file <none>
I20260924 15:25:34.153596 17765 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:34.153667 17765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:34.153855 17765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:34.154889 17765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "d12400c3c4414fb5add7c47aabdabe07"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "5df1d04d4253ed971b1f7c24329352e9"
server_key_iv: "3472cb0903ee919b5fa06ab7ff7b0cac"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.155236 17765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "d12400c3c4414fb5add7c47aabdabe07"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "5df1d04d4253ed971b1f7c24329352e9"
server_key_iv: "3472cb0903ee919b5fa06ab7ff7b0cac"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.156536 17765 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:25:34.157274 17781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:34.157486 17765 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:34.157580 17765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "d12400c3c4414fb5add7c47aabdabe07"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "5df1d04d4253ed971b1f7c24329352e9"
server_key_iv: "3472cb0903ee919b5fa06ab7ff7b0cac"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.157634 17765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:34.172389 17765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:34.174177 17765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:34.174307 17765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:34.178169 17765 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:34259
I20260924 15:25:34.178205 17833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:34259 every 8 connection(s)
I20260924 15:25:34.178568 17765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:25:34.179446 17834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:34.180958 17834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:25:34.181557 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 17765
I20260924 15:25:34.181658 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:25:34.181897 4409 external_mini_cluster.cc:1428] Setting key 77dbfa676879c7bd3135560e18b978c3
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.78.125@KRBTEST.COM; defaulting to no policy
I20260924 15:25:34.191457 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.181685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:34259 (local address 127.0.0.1:54200)
0924 15:25:34.181932 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.181992 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.182485 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.182772 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.182774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.183144 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.183381 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.183387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.184609 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.184610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.185282 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.185284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.185988 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.185989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.186048 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.186568 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.186576 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.188124 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.190111 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.190117 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.190119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.190297 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.190555 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.190555 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.190556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.190629 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.191004 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.191009 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.191063 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.191138 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.191254 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":95,"threads_started":1}
I20260924 15:25:34.191464 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.181917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54200 (local address 127.4.78.126:34259)
0924 15:25:34.182077 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.182081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.182113 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:25:34.182643 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.182645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.182669 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.182773 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.183495 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.184517 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.185361 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.185906 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.186029 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.188225 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.188232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.188233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.188240 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.190042 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.190394 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.190397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.190397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.190484 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.190716 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.190717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.190717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.190933 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.191007 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.191186 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.191316 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260924 15:25:34.192132 17834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.125@KRBTEST.COM" created.
I20260924 15:25:34.192523 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.192412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47440)
0924 15:25:34.192458 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.192500 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:25:34.192663 17784 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
I20260924 15:25:34.193061 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.192974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47456)
0924 15:25:34.193017 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.193040 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:25:34.193416 17834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:42917: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab.
Entry for principal kudu/127.4.78.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab.
I20260924 15:25:34.207978 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.125:42917
--webserver_interface=127.4.78.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab
--principal=kudu/127.4.78.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:34259,127.4.78.125:42917,127.4.78.124:37719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:25:34.242631 17834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } attempt: 1
I20260924 15:25:34.243129 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.242998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47470)
0924 15:25:34.243061 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.243099 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 15:25:34.243577 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.243489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47472)
0924 15:25:34.243537 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.243561 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:25:34.243708 17834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:42917: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
W20260924 15:25:34.288396 17841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:34.288620 17841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:34.288657 17841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:34.289835 17841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:34.289901 17841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:34.289927 17841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:34.289959 17841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:34.289983 17841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:34.291441 17841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:34259,127.4.78.125:42917,127.4.78.124:37719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab
--principal=kudu/127.4.78.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.125:42917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.78.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:34.291708 17841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:34.291957 17841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:34.296627 17850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:34.296622 17848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:34.296622 17847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:34.296859 17841 server_base.cc:1034] running on GCE node
I20260924 15:25:34.297101 17841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:34.297379 17841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:34.298545 17841 hybrid_clock.cc:648] HybridClock initialized: now 1790263534298544 us; error 40 us; skew 500 ppm
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:34.300997 17841 init.cc:377] Logged in from keytab as kudu/127.4.78.125@KRBTEST.COM (short username kudu)
I20260924 15:25:34.301409 17841 webserver.cc:460] Webserver started at http://127.4.78.125:39461/ using document root <none> and password file <none>
I20260924 15:25:34.301645 17841 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:34.301714 17841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:34.301899 17841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:34.302927 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/data/instance:
uuid: "e9a6a12cf870431fa75f904c80a5262e"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "bedd6547c204567cdeb0d9fe381f4177"
server_key_iv: "114f70b3e026a4c284aa1d063e2718e3"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.303282 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/wal/instance:
uuid: "e9a6a12cf870431fa75f904c80a5262e"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "bedd6547c204567cdeb0d9fe381f4177"
server_key_iv: "114f70b3e026a4c284aa1d063e2718e3"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.304570 17841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:25:34.305248 17857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:34.305409 17841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:25:34.305507 17841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/wal
uuid: "e9a6a12cf870431fa75f904c80a5262e"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "bedd6547c204567cdeb0d9fe381f4177"
server_key_iv: "114f70b3e026a4c284aa1d063e2718e3"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.305590 17841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:34.311858 17834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } attempt: 2
I20260924 15:25:34.312312 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.312173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47486)
0924 15:25:34.312239 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.312282 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 15:25:34.312637 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.312557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47494)
0924 15:25:34.312598 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.312620 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:25:34.312763 17834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:42917: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42917: connect: Connection refused (error 111)
I20260924 15:25:34.315594 17841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:34.317668 17841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:34.317898 17841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:34.321664 17841 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.125:42917
I20260924 15:25:34.321727 17909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.125:42917 every 8 connection(s)
I20260924 15:25:34.322085 17841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
I20260924 15:25:34.322435 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 17841
I20260924 15:25:34.322542 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-1/wal/instance
I20260924 15:25:34.322724 4409 external_mini_cluster.cc:1428] Setting key 94f74f6de82e7c56f49af3d412356b5d
I20260924 15:25:34.323061 17910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:34.324637 17910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.78.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.124@KRBTEST.COM" created.
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:34.336123 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.325352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.4.78.126:34259)
0924 15:25:34.325407 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.325410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.325589 (+ 179us) server_negotiation.cc:407] Connection header received
0924 15:25:34.326153 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.326154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.326169 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.326231 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.326824 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.327310 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.328281 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.328484 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.328591 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.333117 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.333119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.333119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.333127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.334679 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.335150 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.335151 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.335152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.335216 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.335464 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.335465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.335465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.335622 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.335695 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.335945 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.336032 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:25:34.336326 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.325318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:34259 (local address 127.0.0.1:54214)
0924 15:25:34.325475 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.325508 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.326052 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.326303 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.326306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.326541 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.326720 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.326724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.327420 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.327421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.328187 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.328189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.328598 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.328599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.328724 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.329227 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.329235 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.333019 (+ 3784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.334776 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.334783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.334785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.335060 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.335295 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.335296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.335296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.335394 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.335689 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.335694 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.335807 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.335921 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.336074 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260924 15:25:34.336851 17910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:25:34.346112 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.337129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47502)
0924 15:25:34.337187 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.337199 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.337245 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.337458 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.337459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.337625 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.337688 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.337690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.338573 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.338574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.339438 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.339441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.339884 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.339886 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.339967 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.340331 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.340337 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.343018 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.344904 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.344905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.344906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.345228 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.345422 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.345422 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.345422 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.345476 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.345782 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.345783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.345881 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.345974 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.346051 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:25:34.346227 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.337165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47502 (local address 127.4.78.125:42917)
0924 15:25:34.337326 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.337330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.337343 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:25:34.337361 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.337363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.337374 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.337433 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.337766 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.338485 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.339522 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.339803 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.339857 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.343162 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.343170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.343171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.343178 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.344831 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.345313 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.345315 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.345316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.345364 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.345526 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.345527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.345527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.345642 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.345737 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.346001 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.346107 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260924 15:25:34.346791 17910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:25:34.347146 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.347037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56822)
0924 15:25:34.347087 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.347129 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:25:34.347288 17858 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
I20260924 15:25:34.347687 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.347593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56832)
0924 15:25:34.347634 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.347657 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:25:34.347821 17910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:37719: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab.
Entry for principal kudu/127.4.78.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab.
I20260924 15:25:34.348884 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.124:37719
--webserver_interface=127.4.78.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab
--principal=kudu/127.4.78.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:34259,127.4.78.125:42917,127.4.78.124:37719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:25:34.397078 17910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } attempt: 1
I20260924 15:25:34.397608 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.397460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56844)
0924 15:25:34.397525 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.397572 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 15:25:34.398002 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.397908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56848)
0924 15:25:34.397952 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.397980 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:25:34.398102 17910 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:37719: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
I20260924 15:25:34.403946 17834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } attempt: 3
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:25:34.411188 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.404325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.4.78.125:42917)
0924 15:25:34.404385 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.404387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.404398 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:25:34.404459 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.404460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.404471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.404536 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.404885 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.405389 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.406124 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.406359 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.406437 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.408459 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.408460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.408460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.408468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.410027 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.410405 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.410407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.410407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.410465 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.410671 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.410671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.410672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.410787 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.410859 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.411021 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.411095 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:25:34.411191 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.404282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47516)
0924 15:25:34.404324 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.404335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.404391 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.404562 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.404563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.404726 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.404804 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.404806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.405473 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.405475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.406031 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.406033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.406456 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.406457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.406562 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.407000 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.407006 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.408373 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.410142 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.410143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.410143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.410318 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.410541 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.410541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.410542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.410603 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.410865 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.410866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.410910 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.410996 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.411094 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 15:25:34.411612 17834 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:25:34.411966 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.411871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56862)
0924 15:25:34.411916 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.411945 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:25:34.412344 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.412242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56868)
0924 15:25:34.412297 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.412325 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 15:25:34.412431 17834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:37719: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
W20260924 15:25:34.430838 17917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:34.431028 17917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:34.431051 17917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:34.432186 17917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:34.432224 17917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:34.432238 17917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:34.432251 17917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:34.432300 17917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:34.433727 17917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:34259,127.4.78.125:42917,127.4.78.124:37719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab
--principal=kudu/127.4.78.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.124:37719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.78.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:34.433995 17917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:34.434219 17917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:34.438867 17922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:34.438862 17925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:34.438894 17923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:34.439339 17917 server_base.cc:1034] running on GCE node
I20260924 15:25:34.439581 17917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:34.439889 17917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:34.441045 17917 hybrid_clock.cc:648] HybridClock initialized: now 1790263534441035 us; error 26 us; skew 500 ppm
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:34.443429 17917 init.cc:377] Logged in from keytab as kudu/127.4.78.124@KRBTEST.COM (short username kudu)
I20260924 15:25:34.443568 17834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } attempt: 1
I20260924 15:25:34.443938 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.443812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56884)
0924 15:25:34.443874 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.443907 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:25:34.443994 17917 webserver.cc:460] Webserver started at http://127.4.78.124:33487/ using document root <none> and password file <none>
I20260924 15:25:34.444245 17917 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:34.444271 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.444186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56896)
0924 15:25:34.444227 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.444251 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 15:25:34.444329 17917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260924 15:25:34.444372 17834 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:37719: Network error: Client connection negotiation failed: client connection to 127.4.78.124:37719: connect: Connection refused (error 111)
I20260924 15:25:34.444466 17917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:34.445503 17917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/data/instance:
uuid: "540e8d6602f64532bc3fccedb8dc4c8b"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "f2632bbfe8957c3c9bb6fd3d0888e4df"
server_key_iv: "0a35c452cf919e1070850db6a002a9fb"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.445917 17917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/wal/instance:
uuid: "540e8d6602f64532bc3fccedb8dc4c8b"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "f2632bbfe8957c3c9bb6fd3d0888e4df"
server_key_iv: "0a35c452cf919e1070850db6a002a9fb"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.447196 17917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:25:34.447906 17932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:34.448081 17917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:25:34.448177 17917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/wal
uuid: "540e8d6602f64532bc3fccedb8dc4c8b"
format_stamp: "Formatted at 2026-09-24 15:25:34 on dist-test-slave-l766"
server_key: "f2632bbfe8957c3c9bb6fd3d0888e4df"
server_key_iv: "0a35c452cf919e1070850db6a002a9fb"
server_key_version: "encryptionkey@0"
I20260924 15:25:34.448263 17917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:34.458825 17917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:34.460748 17917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:34.460901 17917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:34.464897 17917 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.124:37719
I20260924 15:25:34.464946 17985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.124:37719 every 8 connection(s)
I20260924 15:25:34.465304 17917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
I20260924 15:25:34.466173 17986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:34.466271 17910 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } attempt: 2
I20260924 15:25:34.468407 17986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:25:34.473117 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 17917
I20260924 15:25:34.473238 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263382361574-4409-0/minicluster-data/master-2/wal/instance
I20260924 15:25:34.473439 4409 external_mini_cluster.cc:1428] Setting key d8490195c2bf5616b19cd71722a2cef5
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:34.475639 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.466555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56900)
0924 15:25:34.466618 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.466631 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.466703 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.467479 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.467480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.467658 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.467708 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.467711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.468849 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.468851 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.469484 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.469487 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.470167 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.470168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.470223 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.470612 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.470618 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.471885 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.473929 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.473930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.473931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.474172 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.474430 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.474431 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.474431 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.474523 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.475287 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.475289 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.475339 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.475424 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.475516 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:25:34.475849 17988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.466721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.4.78.124:37719)
0924 15:25:34.466867 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.466875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.466897 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:25:34.467301 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.467307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.467349 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.467459 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.467807 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.468739 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.469571 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.470072 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.470186 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.471982 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.471986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.471986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.471995 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.473796 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.474257 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.474260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.474260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.474369 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.474580 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.474581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.474581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.475191 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.475276 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.475499 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.475617 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
I20260924 15:25:34.477059 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.468974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:34259 (local address 127.0.0.1:54228)
0924 15:25:34.469181 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.469218 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.469299 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.469438 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.469441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.469678 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.469737 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.469739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.470436 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.470437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.471297 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.471300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.471699 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.471700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.471777 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.472244 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.472250 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.474002 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.475882 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.475889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.475892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.476147 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.476370 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.476371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.476371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.476436 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.476725 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.476731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.476791 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.476874 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.476967 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
I20260924 15:25:34.477110 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.469030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.4.78.126:34259)
0924 15:25:34.469098 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.469111 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.469282 (+ 171us) server_negotiation.cc:407] Connection header received
0924 15:25:34.469353 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.469354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.469366 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.469421 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.469844 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.470319 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.471388 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.471601 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.471680 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.474098 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.474099 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.474100 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.474109 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.475789 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.476211 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.476212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.476213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.476299 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.476501 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.476502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.476502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.476631 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.476694 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.476922 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.477017 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:25:34.477483 17986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:25:34.478251 17910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e: Bootstrap starting.
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:25:34.479177 17910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:34.479470 17910 log.cc:824] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:34.480347 17910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e: No bootstrap required, opened a new log
I20260924 15:25:34.482750 17910 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.482892 17910 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:34.482932 17910 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a6a12cf870431fa75f904c80a5262e, State: Initialized, Role: FOLLOWER
I20260924 15:25:34.483093 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.473758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56914 (local address 127.4.78.124:37719)
0924 15:25:34.473964 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.473966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.474168 (+ 202us) server_negotiation.cc:407] Connection header received
0924 15:25:34.474188 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.474189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.474201 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.474262 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.475015 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.475699 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.476713 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.476968 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.477030 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.479463 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.479464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.479465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.479471 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.481547 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.481963 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.481964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.481965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.482025 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.482256 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.482256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.482257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.482377 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.482504 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.482899 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.482977 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":100,"threads_started":1}
I20260924 15:25:34.483075 17910 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.483362 17993 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.483425 17910 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:34.483438 17993 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
W20260924 15:25:34.486369 18004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:25:34.486411 18004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:25:34.486624 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.477887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42917 (local address 127.0.0.1:47524)
0924 15:25:34.477926 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.477935 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.477969 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.479886 (+ 1917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.479887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.480054 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.480096 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.480098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.481084 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.481085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.481558 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.481560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.482092 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.482093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.482164 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.482554 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.482559 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.483825 (+ 1266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.485567 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.485568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.485568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.485772 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.485987 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.485988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.485988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.486046 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.486338 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.486339 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.486377 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.486436 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.486517 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 15:25:34.487154 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.477930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.4.78.125:42917)
0924 15:25:34.479736 (+ 1806us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.479738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.479748 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:25:34.479774 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.479775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.479785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.479841 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.480374 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.480991 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.481677 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.482006 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.482093 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.483919 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.483920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.483920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.483928 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.485481 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.485872 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.485873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.485873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.485925 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.486104 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.486105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.486105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.486256 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.486313 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.486991 (+ 678us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.487067 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1786}
I20260924 15:25:34.487576 17986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:34.486281 17841 master_runner.cc:186] Error getting master registration for 127.4.78.126:34259: OK, instance_id { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" instance_seqno: 1790263534176956 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:25:34.489145 17841 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:25:34.492426 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.483893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54238 (local address 127.4.78.126:34259)
0924 15:25:34.483930 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.483933 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.483940 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:25:34.483964 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.483965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.483974 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.484027 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.484674 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.485211 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.486230 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.486527 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.486592 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.489127 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.489129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.489129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.489136 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.490844 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.491241 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.491242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.491243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.491304 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.491530 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.491531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.491532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.491670 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.491845 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.492257 (+ 412us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.492336 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:25:34.494659 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.487835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56928)
0924 15:25:34.487861 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.487871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.487910 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.488028 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.488028 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.488167 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.488200 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.488202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.489072 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.489073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.489731 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.489733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.490248 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.490249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.490299 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.490678 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.490682 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.491929 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.493635 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.493636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.493636 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.493859 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.494041 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.494042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.494043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.494095 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.494346 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.494347 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.494382 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.494449 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.494539 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13}
I20260924 15:25:34.494710 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.487865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.4.78.124:37719)
0924 15:25:34.487896 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.487898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.487906 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:25:34.487964 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.487965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.487973 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.488025 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.488276 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.488985 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.489839 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.490175 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.490247 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.491992 (+ 1745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.491994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.491994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.492001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.493558 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.493925 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.493926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.493926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.493976 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.494176 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.494177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.494177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.494282 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.494354 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.494477 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.494588 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":18}
I20260924 15:25:34.495913 17986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: Bootstrap starting.
I20260924 15:25:34.496631 17986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:34.496999 17986 log.cc:824] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:34.497704 17986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: No bootstrap required, opened a new log
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:25:34.499713 17986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.499872 17986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:34.499912 17986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540e8d6602f64532bc3fccedb8dc4c8b, State: Initialized, Role: FOLLOWER
I20260924 15:25:34.500043 17986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.500366 17986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:34.501899 18007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.502174 18007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:25:34.502575 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.493563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.4.78.125:42917)
0924 15:25:34.493596 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.493597 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.493671 (+ 74us) server_negotiation.cc:407] Connection header received
0924 15:25:34.493748 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.493749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.493766 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.493859 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.494416 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.495115 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.496266 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.496539 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.496617 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.499264 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.499265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.499266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.499273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.501028 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.501463 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.501464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.501464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.501527 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.501777 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.501778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.501779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.502024 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.502170 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.502402 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.502474 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:25:34.502424 17917 master_runner.cc:186] Error getting master registration for 127.4.78.126:34259: OK, instance_id { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" instance_seqno: 1790263534176956 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:25:34.505074 17917 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 15:25:34.505533 18018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:25:34.505582 18018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:25:34.506644 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:25:34.516947 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.508319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56938 (local address 127.4.78.124:37719)
0924 15:25:34.508354 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.508356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.508365 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:25:34.508449 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.508450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.508460 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.508490 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.509324 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.510070 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.511141 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.511457 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.511501 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.513042 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.513043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.513044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.513050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.515564 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.515967 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.515968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.515968 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.516025 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.516229 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.516230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.516230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.516352 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.516537 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.516772 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.516850 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:25:34.517086 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.508493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.4.78.125:42917)
0924 15:25:34.508526 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.508528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.509974 (+ 1446us) server_negotiation.cc:407] Connection header received
0924 15:25:34.510069 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.510071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.510081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.510109 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.510830 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.511533 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.512529 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.512835 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.512873 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.514325 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.514326 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.514326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.514333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.515892 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.516264 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.516265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.516265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.516320 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.516520 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.516521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.516522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.516636 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.516725 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.516924 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.516997 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:25:34.519521 17834 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } attempt: 2
Sep 24 15:25:34 dist-test-slave-l766 krb5kdc[17752](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263534, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:25:34.528558 17836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.519767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:37719 (local address 127.0.0.1:56940)
0924 15:25:34.519807 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:34.519819 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:34.519860 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:34.520065 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:34.520066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:34.520248 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:34.520287 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.520289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.521152 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.521153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.522129 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:34.522132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.522695 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.522696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:34.522756 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.523142 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:34.523147 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:34.524694 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:34.527152 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.527153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.527154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.527459 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.527833 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:34.527835 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:34.527835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:34.527932 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:34.528228 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:34.528229 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:34.528275 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 15:25:34.528341 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:25:34.528460 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 15:25:34.528940 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.519897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.4.78.124:37719)
0924 15:25:34.519936 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.519937 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.519944 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:25:34.519963 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.519964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.519971 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.520026 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.520379 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.521049 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.522286 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.522581 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.522645 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.524797 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.524799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.524799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.524806 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.526995 (+ 2189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.527588 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.527589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.527589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.527675 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.528008 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.528009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.528009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.528142 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.528201 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.528778 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.528856 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:25:34.528996 17837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:34.508789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.4.78.126:34259)
0924 15:25:34.508825 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:34.508827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:34.513266 (+ 4439us) server_negotiation.cc:407] Connection header received
0924 15:25:34.513377 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:34.513378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:34.513389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:34.513418 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:34.518028 (+ 4610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.518761 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.519736 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:34.520033 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:34.520102 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:34.521542 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:34.521543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:34.521543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:34.521550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:34.523434 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.528143 (+ 4709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.528144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.528144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.528217 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:34.528426 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:34.528427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:34.528428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:34.528540 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:34.528618 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:34.528838 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 15:25:34.528907 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:25:34.530480 17834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: Bootstrap starting.
I20260924 15:25:34.531126 17834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:34.531443 17834 log.cc:824] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:34.532289 17834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: No bootstrap required, opened a new log
I20260924 15:25:34.534322 17834 raft_consensus.cc:348] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.534485 17834 raft_consensus.cc:374] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:34.534545 17834 raft_consensus.cc:729] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12400c3c4414fb5add7c47aabdabe07, State: Initialized, Role: FOLLOWER
I20260924 15:25:34.534672 17834 consensus_queue.cc:260] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.535014 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.535079 17834 sys_catalog.cc:564] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:34.535149 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:25:34.539908 17765 master_runner.cc:263] No leader master found from master d12400c3c4414fb5add7c47aabdabe07
I20260924 15:25:34.539955 17765 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 15:25:34.540537 18039 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:25:34.540640 18039 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260924 15:25:34.568538 17863 tablet.cc:2369] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:25:34.675004 17787 tablet.cc:2369] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:25:34.711512 17938 tablet.cc:2369] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:25:34.772042 17993 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:34.772205 17993 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.772455 17993 leader_election.cc:288] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d12400c3c4414fb5add7c47aabdabe07 (127.4.78.126:34259), 540e8d6602f64532bc3fccedb8dc4c8b (127.4.78.124:37719)
I20260924 15:25:34.772763 17959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9a6a12cf870431fa75f904c80a5262e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" is_pre_election: true
I20260924 15:25:34.772768 17808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9a6a12cf870431fa75f904c80a5262e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12400c3c4414fb5add7c47aabdabe07" is_pre_election: true
I20260924 15:25:34.772905 17959 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a6a12cf870431fa75f904c80a5262e in term 0.
I20260924 15:25:34.772938 17808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a6a12cf870431fa75f904c80a5262e in term 0.
I20260924 15:25:34.773128 17858 leader_election.cc:302] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 540e8d6602f64532bc3fccedb8dc4c8b, e9a6a12cf870431fa75f904c80a5262e; no voters:
I20260924 15:25:34.773296 17993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:25:34.773362 17993 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:25:34.773379 17993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:34.774115 17993 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.774266 17993 leader_election.cc:288] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [CANDIDATE]: Term 1 election: Requested vote from peers d12400c3c4414fb5add7c47aabdabe07 (127.4.78.126:34259), 540e8d6602f64532bc3fccedb8dc4c8b (127.4.78.124:37719)
I20260924 15:25:34.774470 17959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9a6a12cf870431fa75f904c80a5262e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540e8d6602f64532bc3fccedb8dc4c8b"
I20260924 15:25:34.774482 17808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9a6a12cf870431fa75f904c80a5262e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d12400c3c4414fb5add7c47aabdabe07"
I20260924 15:25:34.774554 17959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:34.774578 17808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:34.775314 17808 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a6a12cf870431fa75f904c80a5262e in term 1.
I20260924 15:25:34.775362 17959 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a6a12cf870431fa75f904c80a5262e in term 1.
I20260924 15:25:34.775508 17861 leader_election.cc:302] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d12400c3c4414fb5add7c47aabdabe07, e9a6a12cf870431fa75f904c80a5262e; no voters:
I20260924 15:25:34.775617 17993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:34.775812 17993 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [term 1 LEADER]: Becoming Leader. State: Replica: e9a6a12cf870431fa75f904c80a5262e, State: Running, Role: LEADER
I20260924 15:25:34.775921 17993 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } }
I20260924 15:25:34.776352 18041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9a6a12cf870431fa75f904c80a5262e. Latest consensus state: current_term: 1 leader_uuid: "e9a6a12cf870431fa75f904c80a5262e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.776437 18041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:34.776664 18043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:25:34.776893 18043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:25:34.778432 17808 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [term 1 FOLLOWER]: Refusing update from remote peer e9a6a12cf870431fa75f904c80a5262e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:34.778443 17959 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [term 1 FOLLOWER]: Refusing update from remote peer e9a6a12cf870431fa75f904c80a5262e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:34.778781 18041 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:34.778932 17993 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [LEADER]: Connected to new peer: Peer: permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:34.780124 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9a6a12cf870431fa75f904c80a5262e. Latest consensus state: current_term: 1 leader_uuid: "e9a6a12cf870431fa75f904c80a5262e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.780159 18007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9a6a12cf870431fa75f904c80a5262e. Latest consensus state: current_term: 1 leader_uuid: "e9a6a12cf870431fa75f904c80a5262e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.780220 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:25:34.780246 18007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:25:34.781225 18041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9a6a12cf870431fa75f904c80a5262e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.781319 18041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:34.781389 18007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9a6a12cf870431fa75f904c80a5262e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.781471 18007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:25:34.781565 18043 catalog_manager.cc:1347] Generated new cluster ID: de074b24da414aff909bd52133c4598c
I20260924 15:25:34.781612 18043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:25:34.781947 18041 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e9a6a12cf870431fa75f904c80a5262e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.782011 18041 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:34.782346 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e9a6a12cf870431fa75f904c80a5262e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d12400c3c4414fb5add7c47aabdabe07" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34259 } } peers { permanent_uuid: "e9a6a12cf870431fa75f904c80a5262e" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42917 } } peers { permanent_uuid: "540e8d6602f64532bc3fccedb8dc4c8b" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 37719 } } }
I20260924 15:25:34.782418 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:25:34.792541 18043 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:25:34.793244 18043 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:25:34.799460 18043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9a6a12cf870431fa75f904c80a5262e: Generated new TSK 0
I20260924 15:25:34.799649 18043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:25:35.506067 18018 catalog_manager.cc:1259] Loaded cluster ID: de074b24da414aff909bd52133c4598c
I20260924 15:25:35.506137 18018 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: loading cluster ID for follower catalog manager: success
I20260924 15:25:35.506994 18018 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: acquiring CA information for follower catalog manager: success
I20260924 15:25:35.507318 18018 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 540e8d6602f64532bc3fccedb8dc4c8b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:25:35.541111 18039 catalog_manager.cc:1259] Loaded cluster ID: de074b24da414aff909bd52133c4598c
I20260924 15:25:35.541186 18039 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: loading cluster ID for follower catalog manager: success
I20260924 15:25:35.542130 18039 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: acquiring CA information for follower catalog manager: success
I20260924 15:25:35.542454 18039 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d12400c3c4414fb5add7c47aabdabe07: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:25:35.859287 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 17765
I20260924 15:25:35.864370 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 17841
I20260924 15:25:35.869195 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 17917
2026-09-24T15:25:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2981 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:25:35 dist-test-slave-l766 krb5kdc[18059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:25:35 dist-test-slave-l766 krb5kdc[18059](info): set up 2 sockets
Sep 24 15:25:35 dist-test-slave-l766 krb5kdc[18059](info): commencing operation
krb5kdc: starting...
W20260924 15:25:36.905189 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:25:36 dist-test-slave-l766 krb5kdc[18059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:25:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:25:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:25:37.025360 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35367
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:35367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:37.106612 18072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:37.106825 18072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:37.106853 18072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:37.108047 18072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:37.108089 18072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:37.108103 18072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:37.108115 18072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:37.108127 18072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:37.109686 18072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:35367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:37.109946 18072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:37.110193 18072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:37.115116 18077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:37.115104 18072 server_base.cc:1034] running on GCE node
W20260924 15:25:37.115046 18078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:37.115099 18080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:37.115675 18072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:37.115989 18072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:37.117166 18072 hybrid_clock.cc:648] HybridClock initialized: now 1790263537117148 us; error 34 us; skew 500 ppm
Sep 24 15:25:37 dist-test-slave-l766 krb5kdc[18059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263537, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:37.119841 18072 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:25:37.120344 18072 webserver.cc:460] Webserver started at http://127.4.78.126:45499/ using document root <none> and password file <none>
I20260924 15:25:37.120580 18072 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:37.120649 18072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:37.120782 18072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:37.121946 18072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "11b4504fd5124e88bd893f027bb4f8b5"
format_stamp: "Formatted at 2026-09-24 15:25:37 on dist-test-slave-l766"
server_key: "cf14e272851749698a941d6f8fed536f"
server_key_iv: "1d6aa71b250618a2be2cd923e0dbd378"
server_key_version: "encryptionkey@0"
I20260924 15:25:37.122321 18072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "11b4504fd5124e88bd893f027bb4f8b5"
format_stamp: "Formatted at 2026-09-24 15:25:37 on dist-test-slave-l766"
server_key: "cf14e272851749698a941d6f8fed536f"
server_key_iv: "1d6aa71b250618a2be2cd923e0dbd378"
server_key_version: "encryptionkey@0"
I20260924 15:25:37.123781 18072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:25:37.124608 18087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:37.124818 18072 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:25:37.124931 18072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "11b4504fd5124e88bd893f027bb4f8b5"
format_stamp: "Formatted at 2026-09-24 15:25:37 on dist-test-slave-l766"
server_key: "cf14e272851749698a941d6f8fed536f"
server_key_iv: "1d6aa71b250618a2be2cd923e0dbd378"
server_key_version: "encryptionkey@0"
I20260924 15:25:37.125007 18072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:37.134186 18072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:37.136399 18072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:37.136557 18072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:37.140751 18072 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:35367
I20260924 15:25:37.140808 18140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:35367 every 8 connection(s)
I20260924 15:25:37.141163 18072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:25:37.142088 18141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:37.144305 18141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5: Bootstrap starting.
I20260924 15:25:37.144840 18141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:37.145076 18141 log.cc:824] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:37.145877 18141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5: No bootstrap required, opened a new log
I20260924 15:25:37.147503 18141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b4504fd5124e88bd893f027bb4f8b5" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35367 } }
I20260924 15:25:37.147639 18141 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:37.147661 18141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11b4504fd5124e88bd893f027bb4f8b5, State: Initialized, Role: FOLLOWER
I20260924 15:25:37.147805 18141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b4504fd5124e88bd893f027bb4f8b5" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35367 } }
I20260924 15:25:37.147897 18141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:25:37.147933 18141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:25:37.148001 18141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:37.148559 18141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b4504fd5124e88bd893f027bb4f8b5" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35367 } }
I20260924 15:25:37.148679 18141 leader_election.cc:302] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11b4504fd5124e88bd893f027bb4f8b5; no voters:
I20260924 15:25:37.148856 18141 leader_election.cc:288] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:25:37.148906 18144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:37.149087 18144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [term 1 LEADER]: Becoming Leader. State: Replica: 11b4504fd5124e88bd893f027bb4f8b5, State: Running, Role: LEADER
I20260924 15:25:37.149127 18141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:37.149191 18144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b4504fd5124e88bd893f027bb4f8b5" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35367 } }
I20260924 15:25:37.149592 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11b4504fd5124e88bd893f027bb4f8b5. Latest consensus state: current_term: 1 leader_uuid: "11b4504fd5124e88bd893f027bb4f8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b4504fd5124e88bd893f027bb4f8b5" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35367 } } }
I20260924 15:25:37.149724 18145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11b4504fd5124e88bd893f027bb4f8b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11b4504fd5124e88bd893f027bb4f8b5" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35367 } } }
I20260924 15:25:37.149852 18145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:37.149755 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:37.150265 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 18072
I20260924 15:25:37.150377 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:25:37.150565 4409 external_mini_cluster.cc:1428] Setting key e53ec858af3d6343a0be3745a5c77945
W20260924 15:25:37.151001 18160 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:25:37.151057 18160 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:25:37.151374 18153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:25:37.151579 18153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:25:37.153287 18153 catalog_manager.cc:1347] Generated new cluster ID: a738081bd97d48a0abb4eb176962b157
I20260924 15:25:37.153342 18153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:25:37 dist-test-slave-l766 krb5kdc[18059](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:37.160938 18163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:37.151194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.4.78.126:35367)
0924 15:25:37.151385 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:37.151394 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:37.151428 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:25:37.151544 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:37.151553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:37.151621 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:37.151786 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:37.152534 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:37.153781 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:37.154611 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:37.155093 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:37.155550 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:37.157591 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:37.157595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:37.157595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:37.157607 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:37.159293 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:37.159676 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:37.159679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:37.159679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:37.159789 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:37.160004 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:37.160006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:37.160006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:37.160245 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:37.160417 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:37.160517 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:25:37.160771 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260924 15:25:37.161899 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:25:37.171718 18153 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:25:37.172416 18153 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:25:37.184841 18153 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11b4504fd5124e88bd893f027bb4f8b5: Generated new TSK 0
I20260924 15:25:37.185038 18153 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:25:37.217136 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 18072
2026-09-24T15:25:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1347 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:25:37 dist-test-slave-l766 krb5kdc[18168](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:25:37 dist-test-slave-l766 krb5kdc[18168](info): set up 2 sockets
Sep 24 15:25:37 dist-test-slave-l766 krb5kdc[18168](info): commencing operation
krb5kdc: starting...
W20260924 15:25:38.260906 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:25:38 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:25:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:25:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:25:38.380822 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:45845
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:38.461027 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:38.461223 18181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:38.461244 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:38.462436 18181 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:38.462478 18181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:38.462492 18181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:38.462504 18181 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:38.462517 18181 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:38.464015 18181 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:45845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:38.464264 18181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:38.464498 18181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:38.469364 18186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:38.469421 18187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:38.469483 18181 server_base.cc:1034] running on GCE node
W20260924 15:25:38.469421 18189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:38.469944 18181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:38.470239 18181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:38.471412 18181 hybrid_clock.cc:648] HybridClock initialized: now 1790263538471405 us; error 35 us; skew 500 ppm
Sep 24 15:25:38 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:38.473913 18181 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:25:38.474388 18181 webserver.cc:460] Webserver started at http://127.4.78.126:45279/ using document root <none> and password file <none>
I20260924 15:25:38.474627 18181 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:38.474701 18181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:38.474852 18181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:38.475931 18181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e4199575c4ee4f9d8ceb31f90a73be86"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "217aad2d5ab0fab0a548f53981e0dfe0"
server_key_iv: "701553e2b887c159e428a4dfbd112cad"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.476323 18181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e4199575c4ee4f9d8ceb31f90a73be86"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "217aad2d5ab0fab0a548f53981e0dfe0"
server_key_iv: "701553e2b887c159e428a4dfbd112cad"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.477592 18181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:25:38.478598 18196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.478785 18181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:25:38.478886 18181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e4199575c4ee4f9d8ceb31f90a73be86"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "217aad2d5ab0fab0a548f53981e0dfe0"
server_key_iv: "701553e2b887c159e428a4dfbd112cad"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.478968 18181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:38.488057 18181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:38.489794 18181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:38.490018 18181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:38.493702 18181 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:45845
I20260924 15:25:38.493733 18248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:45845 every 8 connection(s)
I20260924 15:25:38.494128 18181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:25:38.495075 18249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:38.495494 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 18181
I20260924 15:25:38.495584 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:25:38.495797 4409 external_mini_cluster.cc:1428] Setting key 0b508707709ad09a8f62df13abcaf5ca
I20260924 15:25:38.498144 18249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Bootstrap starting.
I20260924 15:25:38.498814 18249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:38.499111 18249 log.cc:824] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:38.499816 18249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: No bootstrap required, opened a new log
I20260924 15:25:38.502121 18249 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:38.502250 18249 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:38.502280 18249 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4199575c4ee4f9d8ceb31f90a73be86, State: Initialized, Role: FOLLOWER
I20260924 15:25:38.502441 18249 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:38.502540 18249 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:25:38.502579 18249 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:25:38.502633 18249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 15:25:38 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:38.503320 18249 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:38.503443 18249 leader_election.cc:302] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4199575c4ee4f9d8ceb31f90a73be86; no voters:
I20260924 15:25:38.503599 18249 leader_election.cc:288] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:25:38.503712 18254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:38.503929 18249 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:38.503957 18254 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 1 LEADER]: Becoming Leader. State: Replica: e4199575c4ee4f9d8ceb31f90a73be86, State: Running, Role: LEADER
I20260924 15:25:38.504199 18254 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:38.504915 18256 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4199575c4ee4f9d8ceb31f90a73be86. Latest consensus state: current_term: 1 leader_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } } }
I20260924 15:25:38.504933 18255 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } } }
I20260924 15:25:38.505023 18256 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:38.505035 18255 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: This master's current role is: LEADER
W20260924 15:25:38.506269 18270 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:25:38.506538 18270 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:25:38.507145 18263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:25:38.507357 18263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:25:38.508845 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:38.496328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.4.78.126:45845)
0924 15:25:38.496525 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:38.496535 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:38.496567 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:25:38.497107 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:38.497116 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:38.497176 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:38.497327 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:38.497999 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.499373 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.500185 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.500825 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.500938 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:38.503697 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:38.503702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:38.503702 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:38.503716 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:38.506965 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:38.507385 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:38.507389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:38.507390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:38.507499 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:38.507738 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:38.507739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:38.507740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:38.508045 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:38.508189 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:38.508493 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 15:25:38.508646 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":75,"threads_started":1}
I20260924 15:25:38.508952 18263 catalog_manager.cc:1347] Generated new cluster ID: 6e307147fafe4456a7675f0395260277
I20260924 15:25:38.509022 18263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:25:38.533026 18263 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:25:38.533563 18263 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:25:38.536737 18263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Generated new TSK 0
I20260924 15:25:38.536924 18263 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:25:38.574523 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45845
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:38.655380 18275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:38.655571 18275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:38.655594 18275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:38.656670 18275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:38.656705 18275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:38.656731 18275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:25:38.658128 18275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:38.658301 18275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:38.658510 18275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:38.663398 18281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:38.663509 18280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:38.663518 18284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:38.663983 18275 server_base.cc:1034] running on GCE node
I20260924 15:25:38.664223 18275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:38.664461 18275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:38.665621 18275 hybrid_clock.cc:648] HybridClock initialized: now 1790263538665607 us; error 46 us; skew 500 ppm
Sep 24 15:25:38 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:38.668092 18275 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:25:38.668594 18275 webserver.cc:460] Webserver started at http://127.4.78.65:41213/ using document root <none> and password file <none>
I20260924 15:25:38.668823 18275 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:38.668887 18275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:38.669019 18275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:38.670149 18275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "47d9a98618b14d648b4b363eb4d27b63"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "586ce03806528c826d67cc9b5d08573f"
server_key_iv: "28d099991830cb6a8231dcf8454f5eb2"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.670521 18275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "47d9a98618b14d648b4b363eb4d27b63"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "586ce03806528c826d67cc9b5d08573f"
server_key_iv: "28d099991830cb6a8231dcf8454f5eb2"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.671837 18275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:38.672583 18291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.672775 18275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:25:38.672892 18275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "47d9a98618b14d648b4b363eb4d27b63"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "586ce03806528c826d67cc9b5d08573f"
server_key_iv: "28d099991830cb6a8231dcf8454f5eb2"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.672976 18275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:38.700181 18275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:38.702214 18275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:38.702390 18275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:38.702638 18275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:38.702965 18275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:38.703022 18275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.703063 18275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:38.703111 18275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.709134 18275 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:43829
I20260924 15:25:38.709197 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:43829 every 8 connection(s)
I20260924 15:25:38.709506 18275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:25:38 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:38.718092 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:38.710217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:56569 (local address 127.4.78.126:45845)
0924 15:25:38.710281 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:38.710283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:38.710465 (+ 182us) server_negotiation.cc:407] Connection header received
0924 15:25:38.710933 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:38.710934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:38.710943 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:38.710999 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:38.711920 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.712448 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.713411 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.713611 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.713665 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:38.715737 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:38.715738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:38.715739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:38.715744 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:38.716925 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:38.717277 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:38.717278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:38.717278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:38.717330 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:38.717534 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:38.717534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:38.717534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:38.717631 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:38.717680 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:38.717959 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 15:25:38.718027 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:25:38.718282 18407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:38.710164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:56569)
0924 15:25:38.710331 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:38.710374 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:38.710848 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:38.711075 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:38.711078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:38.711638 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:38.711828 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:38.711832 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.712528 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.712529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:38.713322 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:38.713324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.713688 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.713689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:38.713789 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:38.714322 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:38.714339 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:38.715655 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:38.716996 (+ 1341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:38.717003 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:38.717004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:38.717214 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:38.717386 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:38.717387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:38.717387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:38.717478 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:38.717685 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:38.717691 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:38.717800 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 15:25:38.717942 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 15:25:38.718056 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260924 15:25:38.718835 18405 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45845
I20260924 15:25:38.718940 18405 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:38.719111 18405 heartbeater.cc:511] Master 127.4.78.126:45845 requested a full tablet report, sending...
I20260924 15:25:38.719429 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 18275
I20260924 15:25:38.719552 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:25:38.719719 18213 ts_manager.cc:194] Registered new tserver with Master: 47d9a98618b14d648b4b363eb4d27b63 (127.4.78.65:43829)
I20260924 15:25:38.719756 4409 external_mini_cluster.cc:1428] Setting key 7246ca122c78a6a8474de6b177227d15
I20260924 15:25:38.720628 18213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:56569
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:25:38.745481 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45845
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:38.825112 18410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:38.825340 18410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:38.825377 18410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:38.826553 18410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:38.826620 18410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:38.826660 18410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:25:38.828133 18410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:38.828387 18410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:38.828626 18410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:38.833696 18415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:38.833729 18416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:38.833915 18410 server_base.cc:1034] running on GCE node
W20260924 15:25:38.833702 18418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:38.834172 18410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:38.834481 18410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:38.835635 18410 hybrid_clock.cc:648] HybridClock initialized: now 1790263538835617 us; error 35 us; skew 500 ppm
Sep 24 15:25:38 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:38.838023 18410 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:25:38.838490 18410 webserver.cc:460] Webserver started at http://127.4.78.66:36171/ using document root <none> and password file <none>
I20260924 15:25:38.838726 18410 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:38.838792 18410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:38.838934 18410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:38.839942 18410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "7bc63a4e79a4434293936c0a80a68317"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "ec95fae3d92b97fcef267c30d2eb7800"
server_key_iv: "ae084687825de3aa74f29ef857fa6a01"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.840312 18410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "7bc63a4e79a4434293936c0a80a68317"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "ec95fae3d92b97fcef267c30d2eb7800"
server_key_iv: "ae084687825de3aa74f29ef857fa6a01"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.841607 18410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:25:38.842417 18425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.842582 18410 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.842679 18410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "7bc63a4e79a4434293936c0a80a68317"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "ec95fae3d92b97fcef267c30d2eb7800"
server_key_iv: "ae084687825de3aa74f29ef857fa6a01"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.842761 18410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:38.859249 18410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:38.861562 18410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:38.861675 18410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:38.861898 18410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:38.862154 18410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:38.862192 18410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.862214 18410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:38.862226 18410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.867707 18410 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:34287
I20260924 15:25:38.867791 18538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:34287 every 8 connection(s)
I20260924 15:25:38.868086 18410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:25:38.870044 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 18410
I20260924 15:25:38.870178 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:25:38.870396 4409 external_mini_cluster.cc:1428] Setting key c6bfd0c9f301bdd6c50c561af8c1522a
Sep 24 15:25:38 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
I20260924 15:25:38.878492 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:38.868862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:56663 (local address 127.4.78.126:45845)
0924 15:25:38.868931 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:38.868933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:38.869142 (+ 209us) server_negotiation.cc:407] Connection header received
0924 15:25:38.869671 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:38.869672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:38.869685 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:38.869742 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:38.870941 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.871442 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.872602 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.872800 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.872858 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:38.875358 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:38.875359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:38.875361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:38.875374 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:38.876985 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:38.877439 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:38.877441 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:38.877441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:38.877502 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:38.877773 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:38.877774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:38.877774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:38.877949 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:38.878023 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:38.878314 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 15:25:38.878399 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:25:38.878717 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:38.868849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:56663)
0924 15:25:38.869014 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:38.869047 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:38.869563 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:38.869836 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:38.869841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:38.870657 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:38.870843 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:38.870848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.871537 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.871538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:38.872511 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:38.872514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:38.872884 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:38.872885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:38.873036 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:38.873524 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:38.873534 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:38.875232 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:38.877079 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:38.877085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:38.877088 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:38.877343 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:38.877572 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:38.877573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:38.877574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:38.877697 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:38.878026 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:38.878033 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:38.878178 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 15:25:38.878291 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 15:25:38.878452 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260924 15:25:38.879212 18539 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45845
I20260924 15:25:38.879310 18539 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:38.879457 18539 heartbeater.cc:511] Master 127.4.78.126:45845 requested a full tablet report, sending...
I20260924 15:25:38.879841 18213 ts_manager.cc:194] Registered new tserver with Master: 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
I20260924 15:25:38.880270 18213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:56663
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:25:38.896361 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45845
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:38.976078 18544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:38.976292 18544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:38.976331 18544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:38.977517 18544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:38.977577 18544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:38.977622 18544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:25:38.979070 18544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:38.979329 18544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:38.979544 18544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:38.984573 18550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:38.984592 18552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:38.984639 18544 server_base.cc:1034] running on GCE node
W20260924 15:25:38.984588 18549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:38.984984 18544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:38.985183 18544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:38.986362 18544 hybrid_clock.cc:648] HybridClock initialized: now 1790263538986346 us; error 26 us; skew 500 ppm
Sep 24 15:25:38 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:38.988862 18544 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:25:38.989255 18544 webserver.cc:460] Webserver started at http://127.4.78.67:33803/ using document root <none> and password file <none>
I20260924 15:25:38.989459 18544 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:38.989499 18544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:38.989584 18544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:38.990643 18544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "6f562ebd694a4a9f9bba64fc9040eb4c"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "43a0a7f021b8e1bc977940e0bd16471b"
server_key_iv: "130b665e15434b384cd5b7346a27b968"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.991016 18544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "6f562ebd694a4a9f9bba64fc9040eb4c"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "43a0a7f021b8e1bc977940e0bd16471b"
server_key_iv: "130b665e15434b384cd5b7346a27b968"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.992249 18544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:25:38.992940 18559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:38.993139 18544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:25:38.993208 18544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "6f562ebd694a4a9f9bba64fc9040eb4c"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "43a0a7f021b8e1bc977940e0bd16471b"
server_key_iv: "130b665e15434b384cd5b7346a27b968"
server_key_version: "encryptionkey@0"
I20260924 15:25:38.993258 18544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:39.008518 18544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:39.010907 18544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:39.011057 18544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:39.011301 18544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:39.011622 18544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:39.011656 18544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.011677 18544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:39.011722 18544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.017477 18544 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:45849
I20260924 15:25:39.017554 18674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:45849 every 8 connection(s)
I20260924 15:25:39.017910 18544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:25:39.020913 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 18544
I20260924 15:25:39.021025 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:25:39.021190 4409 external_mini_cluster.cc:1428] Setting key 698a8dda0b92cb96bd536aca973c6d31
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263538, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:39.026955 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.018698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:41085 (local address 127.4.78.126:45845)
0924 15:25:39.018740 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.018744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.018936 (+ 192us) server_negotiation.cc:407] Connection header received
0924 15:25:39.019416 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.019417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.019427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.019484 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:39.020371 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.020860 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.022256 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.022456 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.022542 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.024540 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:39.024542 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:39.024542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:39.024549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:39.025776 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.026190 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.026191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.026191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.026242 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.026441 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.026442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.026442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.026542 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:39.026593 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.026825 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.026891 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:25:39.027108 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.018656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:41085)
0924 15:25:39.018814 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.018854 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.019335 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.019535 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.019539 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.020099 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:39.020290 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.020294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.020963 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.020964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.022171 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.022174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.022553 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.022554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.022683 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.023133 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:39.023142 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:39.024439 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:39.025911 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.025917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.025919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.026128 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.026307 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.026307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.026307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.026391 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.026597 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:39.026602 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:39.026713 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.026802 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.026914 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260924 15:25:39.027583 18675 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45845
I20260924 15:25:39.027667 18675 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:39.027861 18675 heartbeater.cc:511] Master 127.4.78.126:45845 requested a full tablet report, sending...
I20260924 15:25:39.028228 18213 ts_manager.cc:194] Registered new tserver with Master: 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849)
I20260924 15:25:39.028599 18213 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:41085
I20260924 15:25:39.032975 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:25:39.040228 18252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.034339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.4.78.126:45845)
0924 15:25:39.034381 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.034384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.034509 (+ 125us) server_negotiation.cc:407] Connection header received
0924 15:25:39.034545 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.034545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.034554 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.034609 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:39.035225 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.035674 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.036396 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.036657 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.036759 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.037908 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:39.037909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:39.037910 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:39.037917 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:39.039003 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.039366 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.039367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.039368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.039422 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.039625 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.039626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.039627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.039772 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:39.039879 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.040006 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.040082 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:25:39.041718 18213 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:25:39.043362 18213 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:25:39.049863 18689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.045623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:45849 (local address 127.0.0.1:56230)
0924 15:25:39.045991 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.046002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.046054 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.046443 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.046443 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.046445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.046491 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.046492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.047494 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.047495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.048649 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.048652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.049509 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.049510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.049593 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.049616 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.049692 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.049747 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":73,"threads_started":1}
I20260924 15:25:39.049898 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.045719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.4.78.67:45849)
0924 15:25:39.046273 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.046277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.046287 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:25:39.046320 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.046322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.046338 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.046566 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.046640 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.047388 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.048767 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.049417 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.049492 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.049549 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.049720 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.049776 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":125,"threads_started":1}
I20260924 15:25:39.050081 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.045621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43829 (local address 127.0.0.1:45346)
0924 15:25:39.045775 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.045839 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.045906 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.046317 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.046319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.046322 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.046418 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.046420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.047766 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.047767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.048924 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.048926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.049852 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.049853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.049922 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.049926 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.049961 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.050009 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":23,"thread_start_us":93,"threads_started":1}
I20260924 15:25:39.050374 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.045965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.4.78.65:43829)
0924 15:25:39.046136 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.046141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.046153 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:25:39.046197 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.046199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.046216 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.046306 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.046481 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.047648 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.049002 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.049738 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.050136 (+ 398us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.050183 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.050204 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.050276 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260924 15:25:39.050966 18609 tablet_service.cc:1467] Processing CreateTablet for tablet 65fe8d6803d74686b80e7a23e9318085 (DEFAULT_TABLE table=test-table [id=283b2fa90e124d0196de6aa65308e1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:39.051013 18339 tablet_service.cc:1467] Processing CreateTablet for tablet 65fe8d6803d74686b80e7a23e9318085 (DEFAULT_TABLE table=test-table [id=283b2fa90e124d0196de6aa65308e1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:39.051317 18609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fe8d6803d74686b80e7a23e9318085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:39.051328 18339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fe8d6803d74686b80e7a23e9318085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:39.053840 18692 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: Bootstrap starting.
I20260924 15:25:39.053884 18693 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Bootstrap starting.
I20260924 15:25:39.054632 18692 tablet_bootstrap.cc:654] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:39.054656 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.045866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:34287 (local address 127.0.0.1:36872)
0924 15:25:39.046143 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.046153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.046212 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.046724 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.046725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.046727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.046761 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.046762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.047982 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.047983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.049725 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.049727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.054381 (+ 4654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.054382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.054450 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.054455 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.054516 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.054573 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":73,"threads_started":1}
I20260924 15:25:39.054720 18693 tablet_bootstrap.cc:654] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:39.054719 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.045979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.4.78.66:34287)
0924 15:25:39.046164 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.046169 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.046203 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:25:39.046541 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.046544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.046561 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.046644 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.046847 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.047874 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.049814 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.054203 (+ 4389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.054282 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.054344 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.054553 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.054607 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260924 15:25:39.055282 18693 log.cc:824] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:39.055469 18692 log.cc:824] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:39.055457 18473 tablet_service.cc:1467] Processing CreateTablet for tablet 65fe8d6803d74686b80e7a23e9318085 (DEFAULT_TABLE table=test-table [id=283b2fa90e124d0196de6aa65308e1a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:39.055800 18473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fe8d6803d74686b80e7a23e9318085. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:39.056095 18693 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: No bootstrap required, opened a new log
I20260924 15:25:39.056218 18693 ts_tablet_manager.cc:1397] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:25:39.056283 18692 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: No bootstrap required, opened a new log
I20260924 15:25:39.056354 18692 ts_tablet_manager.cc:1397] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:25:39.058282 18693 raft_consensus.cc:348] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.058282 18692 raft_consensus.cc:348] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.058463 18692 raft_consensus.cc:374] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:39.058525 18692 raft_consensus.cc:729] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f562ebd694a4a9f9bba64fc9040eb4c, State: Initialized, Role: FOLLOWER
I20260924 15:25:39.058511 18696 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Bootstrap starting.
I20260924 15:25:39.058728 18693 raft_consensus.cc:374] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:39.058693 18692 consensus_queue.cc:260] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.058789 18693 raft_consensus.cc:729] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d9a98618b14d648b4b363eb4d27b63, State: Initialized, Role: FOLLOWER
I20260924 15:25:39.058919 18693 consensus_queue.cc:260] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.058988 18692 ts_tablet_manager.cc:1428] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:25:39.059099 18675 heartbeater.cc:503] Master 127.4.78.126:45845 was elected leader, sending a full tablet report...
I20260924 15:25:39.059228 18696 tablet_bootstrap.cc:654] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:39.059494 18405 heartbeater.cc:503] Master 127.4.78.126:45845 was elected leader, sending a full tablet report...
I20260924 15:25:39.059494 18693 ts_tablet_manager.cc:1428] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:25:39.059783 18696 log.cc:824] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:39.060583 18696 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: No bootstrap required, opened a new log
I20260924 15:25:39.060652 18696 ts_tablet_manager.cc:1397] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:25:39.061975 18696 raft_consensus.cc:348] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.062103 18696 raft_consensus.cc:374] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:39.062124 18696 raft_consensus.cc:729] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc63a4e79a4434293936c0a80a68317, State: Initialized, Role: FOLLOWER
I20260924 15:25:39.062224 18696 consensus_queue.cc:260] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.062484 18696 ts_tablet_manager.cc:1428] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:25:39.062510 18539 heartbeater.cc:503] Master 127.4.78.126:45845 was elected leader, sending a full tablet report...
W20260924 15:25:39.118645 18540 tablet.cc:2369] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:25:39.208451 18697 raft_consensus.cc:482] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:39.208618 18697 raft_consensus.cc:504] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.209092 18697 leader_election.cc:288] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47d9a98618b14d648b4b363eb4d27b63 (127.4.78.65:43829), 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
W20260924 15:25:39.210242 18406 tablet.cc:2369] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:25:39.213096 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.209328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43829 (local address 127.4.78.67:49605)
0924 15:25:39.209396 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.209409 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.209453 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.209642 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.209643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.209645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.209755 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.209757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.210785 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.210786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.212223 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.212227 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.212835 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.212837 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.212919 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.212922 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.212979 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.213020 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":60}
I20260924 15:25:39.213096 18690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.209413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:49605 (local address 127.4.78.65:43829)
0924 15:25:39.209463 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.209466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.209482 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:25:39.209517 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.209519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.209533 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.209617 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.209849 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.210637 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.212330 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.212747 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.212834 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.212844 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.212994 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.213033 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:25:39.213618 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.209430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:37629 (local address 127.4.78.66:34287)
0924 15:25:39.209496 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.209499 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.209707 (+ 208us) server_negotiation.cc:407] Connection header received
0924 15:25:39.209754 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.209755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.209769 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.209841 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.210046 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.210742 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.212435 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.213125 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.213204 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.213214 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.213513 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.213555 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:25:39.213675 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fe8d6803d74686b80e7a23e9318085" candidate_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d9a98618b14d648b4b363eb4d27b63" is_pre_election: true
I20260924 15:25:39.213852 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.209335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:34287 (local address 127.4.78.67:37629)
0924 15:25:39.209628 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.209639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.209702 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.209876 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.209877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.209878 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.209940 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.209942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.210836 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.210837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.212288 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.212292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.213347 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.213348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.213421 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.213426 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.213695 (+ 269us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.213744 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":120,"thread_start_us":97,"threads_started":1}
I20260924 15:25:39.213866 18359 raft_consensus.cc:2454] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f562ebd694a4a9f9bba64fc9040eb4c in term 0.
I20260924 15:25:39.214100 18563 leader_election.cc:302] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d9a98618b14d648b4b363eb4d27b63, 6f562ebd694a4a9f9bba64fc9040eb4c; no voters:
I20260924 15:25:39.214150 18493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fe8d6803d74686b80e7a23e9318085" candidate_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc63a4e79a4434293936c0a80a68317" is_pre_election: true
I20260924 15:25:39.214229 18697 raft_consensus.cc:2781] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:25:39.214286 18493 raft_consensus.cc:2454] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f562ebd694a4a9f9bba64fc9040eb4c in term 0.
I20260924 15:25:39.214298 18697 raft_consensus.cc:482] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:25:39.214329 18697 raft_consensus.cc:3037] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:39.215317 18697 raft_consensus.cc:504] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.215457 18697 leader_election.cc:288] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [CANDIDATE]: Term 1 election: Requested vote from peers 47d9a98618b14d648b4b363eb4d27b63 (127.4.78.65:43829), 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
I20260924 15:25:39.215642 18493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fe8d6803d74686b80e7a23e9318085" candidate_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc63a4e79a4434293936c0a80a68317"
I20260924 15:25:39.215756 18493 raft_consensus.cc:3037] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:39.215876 18359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fe8d6803d74686b80e7a23e9318085" candidate_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47d9a98618b14d648b4b363eb4d27b63"
I20260924 15:25:39.215960 18359 raft_consensus.cc:3037] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:39.216708 18493 raft_consensus.cc:2454] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f562ebd694a4a9f9bba64fc9040eb4c in term 1.
I20260924 15:25:39.216881 18560 leader_election.cc:302] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f562ebd694a4a9f9bba64fc9040eb4c, 7bc63a4e79a4434293936c0a80a68317; no voters:
I20260924 15:25:39.216955 18359 raft_consensus.cc:2454] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f562ebd694a4a9f9bba64fc9040eb4c in term 1.
I20260924 15:25:39.216979 18697 raft_consensus.cc:2781] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:39.217164 18697 raft_consensus.cc:686] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 LEADER]: Becoming Leader. State: Replica: 6f562ebd694a4a9f9bba64fc9040eb4c, State: Running, Role: LEADER
I20260924 15:25:39.217262 18697 consensus_queue.cc:237] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.218149 18212 catalog_manager.cc:5495] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67). New cstate: current_term: 1 leader_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } health_report { overall_health: UNKNOWN } } }
I20260924 15:25:39.234130 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.231643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56246 (local address 127.4.78.67:45849)
0924 15:25:39.231715 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.231720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.231729 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:25:39.231759 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.231760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.231777 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.231848 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:25:39.231973 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.232430 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.233152 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.233388 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.233461 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.233646 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:25:39.233754 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:25:39.233861 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.234035 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.234056 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 15:25:39.235980 18359 raft_consensus.cc:1261] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Refusing update from remote peer 6f562ebd694a4a9f9bba64fc9040eb4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:39.236030 18493 raft_consensus.cc:1261] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 1 FOLLOWER]: Refusing update from remote peer 6f562ebd694a4a9f9bba64fc9040eb4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:39.236361 18697 consensus_queue.cc:1035] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:39.236495 18702 consensus_queue.cc:1035] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:39.238996 18707 mvcc.cc:205] Tried to move back new op lower bound from 7332919456707923968 to 7332919456634486784. Current Snapshot: MvccSnapshot[applied={T|T < 7332919456707923968}]
I20260924 15:25:39.239116 18706 mvcc.cc:205] Tried to move back new op lower bound from 7332919456707923968 to 7332919456634486784. Current Snapshot: MvccSnapshot[applied={T|T < 7332919456707923968}]
I20260924 15:25:39.239233 18705 mvcc.cc:205] Tried to move back new op lower bound from 7332919456707923968 to 7332919456634486784. Current Snapshot: MvccSnapshot[applied={T|T < 7332919456707923968}]
I20260924 15:25:39.241173 18212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:25:39.241317 18212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:25:39.242080 18212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Sending DeleteTablet for 3 replicas of tablet 65fe8d6803d74686b80e7a23e9318085
I20260924 15:25:39.242414 18339 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fe8d6803d74686b80e7a23e9318085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:39 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:45346
I20260924 15:25:39.242460 18609 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fe8d6803d74686b80e7a23e9318085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:39 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:56230
I20260924 15:25:39.242462 18473 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fe8d6803d74686b80e7a23e9318085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:39 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:36872
I20260924 15:25:39.242619 18715 tablet_replica.cc:330] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: stopping tablet replica
I20260924 15:25:39.242712 18715 raft_consensus.cc:2229] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:25:39.242807 18715 raft_consensus.cc:2258] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:39.242802 18717 tablet_replica.cc:330] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: stopping tablet replica
I20260924 15:25:39.242805 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 18275
I20260924 15:25:39.243067 18716 tablet_replica.cc:330] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: stopping tablet replica
I20260924 15:25:39.243507 18716 raft_consensus.cc:2229] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:25:39.243618 18716 raft_consensus.cc:2258] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:39.244238 18716 ts_tablet_manager.cc:1905] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:39.245769 18716 ts_tablet_manager.cc:1918] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:39.245884 18716 log.cc:1196] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/65fe8d6803d74686b80e7a23e9318085
I20260924 15:25:39.242892 18717 raft_consensus.cc:2229] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:25:39.246124 18717 raft_consensus.cc:2258] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:39.246169 18716 ts_tablet_manager.cc:1939] T 65fe8d6803d74686b80e7a23e9318085 P 7bc63a4e79a4434293936c0a80a68317: Deleting consensus metadata
I20260924 15:25:39.246632 18717 ts_tablet_manager.cc:1905] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:39.246698 18197 catalog_manager.cc:4841] TS 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287): tablet 65fe8d6803d74686b80e7a23e9318085 (table test-table [id=283b2fa90e124d0196de6aa65308e1a3]) successfully deleted
I20260924 15:25:39.248268 18717 ts_tablet_manager.cc:1918] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:39.248361 18717 log.cc:1196] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/65fe8d6803d74686b80e7a23e9318085
I20260924 15:25:39.248612 18717 ts_tablet_manager.cc:1939] T 65fe8d6803d74686b80e7a23e9318085 P 6f562ebd694a4a9f9bba64fc9040eb4c: Deleting consensus metadata
I20260924 15:25:39.248991 18197 catalog_manager.cc:4841] TS 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849): tablet 65fe8d6803d74686b80e7a23e9318085 (table test-table [id=283b2fa90e124d0196de6aa65308e1a3]) successfully deleted
W20260924 15:25:39.249573 18200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:43829 (error 108)
I20260924 15:25:39.249796 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 18410
I20260924 15:25:39.250855 18686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.250740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43829 (local address 127.0.0.1:45362)
0924 15:25:39.250787 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.250822 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:43829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:25:39.250964 18200 catalog_manager.cc:4584] TS 47d9a98618b14d648b4b363eb4d27b63 (127.4.78.65:43829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65fe8d6803d74686b80e7a23e9318085: Network error: Client connection negotiation failed: client connection to 127.4.78.65:43829: connect: Connection refused (error 111)
I20260924 15:25:39.255224 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 18544
I20260924 15:25:39.260316 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 18181
I20260924 15:25:39.265748 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:45845
--webserver_interface=127.4.78.126
--webserver_port=45279
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:39.346702 18719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:39.346907 18719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:39.346930 18719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:39.348007 18719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:39.348042 18719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:39.348055 18719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:39.348067 18719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:39.348079 18719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:39.349471 18719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:45845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:39.349655 18719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:39.349839 18719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:39.358333 18726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:39.358385 18725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:39.358358 18728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:39.358606 18719 server_base.cc:1034] running on GCE node
I20260924 15:25:39.358902 18719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:39.359261 18719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:39.360427 18719 hybrid_clock.cc:648] HybridClock initialized: now 1790263539360425 us; error 38 us; skew 500 ppm
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:39.367408 18719 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:25:39.367913 18719 webserver.cc:460] Webserver started at http://127.4.78.126:45279/ using document root <none> and password file <none>
I20260924 15:25:39.368158 18719 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:39.368284 18719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:39.369741 18719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:25:39.370574 18735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.370805 18719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:39.370929 18719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e4199575c4ee4f9d8ceb31f90a73be86"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "217aad2d5ab0fab0a548f53981e0dfe0"
server_key_iv: "701553e2b887c159e428a4dfbd112cad"
server_key_version: "encryptionkey@0"
I20260924 15:25:39.371379 18719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:39.389750 18719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:39.391774 18719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:39.391939 18719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:39.396108 18719 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:45845
I20260924 15:25:39.396142 18787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:45845 every 8 connection(s)
I20260924 15:25:39.396528 18719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:25:39.398207 18788 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:25:39.398834 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Bootstrap starting.
I20260924 15:25:39.400705 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 18719
I20260924 15:25:39.400997 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:43829
--local_ip_for_outbound_sockets=127.4.78.65
--tserver_master_addrs=127.4.78.126:45845
--webserver_port=41213
--webserver_interface=127.4.78.65
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:25:39.407073 18788 log.cc:824] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:39.408959 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:25:39.409281 18788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Bootstrap complete.
I20260924 15:25:39.411608 18788 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:39.411863 18788 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4199575c4ee4f9d8ceb31f90a73be86, State: Initialized, Role: FOLLOWER
I20260924 15:25:39.412068 18788 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:39.412168 18788 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:25:39.412226 18788 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:25:39.412276 18788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:25:39.413291 18788 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:39.413410 18788 leader_election.cc:302] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4199575c4ee4f9d8ceb31f90a73be86; no voters:
I20260924 15:25:39.413607 18788 leader_election.cc:288] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:25:39.413645 18792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:25:39.413748 18792 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 2 LEADER]: Becoming Leader. State: Replica: e4199575c4ee4f9d8ceb31f90a73be86, State: Running, Role: LEADER
I20260924 15:25:39.413861 18792 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:39.413906 18788 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:39.414109 18792 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4199575c4ee4f9d8ceb31f90a73be86. Latest consensus state: current_term: 2 leader_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } } }
I20260924 15:25:39.414183 18792 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:39.414114 18793 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } } }
I20260924 15:25:39.414415 18793 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:39.415762 18808 catalog_manager.cc:1259] Loaded cluster ID: 6e307147fafe4456a7675f0395260277
I20260924 15:25:39.415812 18808 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: loading cluster ID for follower catalog manager: success
W20260924 15:25:39.416002 18808 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260924 15:25:39.416043 18808 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260924 15:25:39.416083 18796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:25:39.416401 18796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:25:39.416495 18796 catalog_manager.cc:1259] Loaded cluster ID: 6e307147fafe4456a7675f0395260277
I20260924 15:25:39.416527 18796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:25:39.416675 18796 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 15:25:39.420572 18796 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 15:25:39.423081 18796 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:25:39.423305 18796 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Loaded TSK: 0
I20260924 15:25:39.423544 18796 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:25:39.485144 18790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:39.485417 18790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:39.485459 18790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:39.486634 18790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:39.486707 18790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:39.486752 18790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:25:39.488197 18790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:43829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=41213
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:39.488453 18790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:39.488691 18790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:39.494344 18816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:39.494375 18818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:39.494632 18815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:39.495041 18790 server_base.cc:1034] running on GCE node
I20260924 15:25:39.495249 18790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:39.495486 18790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:39.496676 18790 hybrid_clock.cc:648] HybridClock initialized: now 1790263539496651 us; error 38 us; skew 500 ppm
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:39.499354 18790 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:25:39.499758 18790 webserver.cc:460] Webserver started at http://127.4.78.65:41213/ using document root <none> and password file <none>
I20260924 15:25:39.499984 18790 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:39.500063 18790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:39.501494 18790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:25:39.502269 18825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.502439 18790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:25:39.502543 18790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "47d9a98618b14d648b4b363eb4d27b63"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "586ce03806528c826d67cc9b5d08573f"
server_key_iv: "28d099991830cb6a8231dcf8454f5eb2"
server_key_version: "encryptionkey@0"
I20260924 15:25:39.502874 18790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:39.524405 18790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:39.526690 18790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:39.526853 18790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:39.527091 18790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:39.527595 18832 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:25:39.529177 18790 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:25:39.529228 18790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:25:39.529265 18790 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:25:39.529989 18790 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:25:39.530035 18790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:39.530086 18832 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Bootstrap starting.
I20260924 15:25:39.537729 18790 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:43829
I20260924 15:25:39.538278 18790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:25:39.540932 18832 log.cc:824] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:39.541291 18939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:43829 every 8 connection(s)
I20260924 15:25:39.543592 18832 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:25:39.543957 18832 tablet_bootstrap.cc:492] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Bootstrap complete.
I20260924 15:25:39.545197 18832 ts_tablet_manager.cc:1397] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.011s
I20260924 15:25:39.546443 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 18790
I20260924 15:25:39.546730 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:34287
--local_ip_for_outbound_sockets=127.4.78.66
--tserver_master_addrs=127.4.78.126:45845
--webserver_port=36171
--webserver_interface=127.4.78.66
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:25:39.546744 18832 raft_consensus.cc:348] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.547382 18832 raft_consensus.cc:729] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d9a98618b14d648b4b363eb4d27b63, State: Initialized, Role: FOLLOWER
I20260924 15:25:39.547586 18832 consensus_queue.cc:260] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } } peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } }
I20260924 15:25:39.547919 18832 ts_tablet_manager.cc:1428] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:39.553395 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.539128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:59155 (local address 127.4.78.126:45845)
0924 15:25:39.541586 (+ 2458us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.541595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.541639 (+ 44us) server_negotiation.cc:407] Connection header received
0924 15:25:39.541784 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.541792 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.541879 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.542027 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:39.542937 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.544314 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.545851 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.546418 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.546523 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.549261 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:39.549265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:39.549266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:39.549276 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:39.551367 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.551895 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.551898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.551898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.552005 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.552335 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.552336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.552336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.552600 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:39.552671 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.553071 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.553212 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2403,"thread_start_us":100,"threads_started":1}
I20260924 15:25:39.553464 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.539003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:59155)
0924 15:25:39.539271 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.539312 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.539864 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.542083 (+ 2219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.542088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.542489 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:39.542778 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.542787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.544444 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.544445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.545731 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.545734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.546613 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.546614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.546755 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.547346 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:39.547356 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:39.549145 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:39.551456 (+ 2311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.551464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.551466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.551798 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.552094 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.552095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.552095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.552256 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.552690 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:39.552698 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:39.552878 (+ 180us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.553001 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.553201 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260924 15:25:39.554730 18940 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45845
I20260924 15:25:39.554836 18940 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:39.555032 18940 heartbeater.cc:511] Master 127.4.78.126:45845 requested a full tablet report, sending...
I20260924 15:25:39.555881 18752 ts_manager.cc:194] Registered new tserver with Master: 47d9a98618b14d648b4b363eb4d27b63 (127.4.78.65:43829)
I20260924 15:25:39.556994 18752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:59155
I20260924 15:25:39.557899 18940 heartbeater.cc:503] Master 127.4.78.126:45845 was elected leader, sending a full tablet report...
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.65@KRBTEST.COM
I20260924 15:25:39.564042 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.556431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43829 (local address 127.0.0.1:45368)
0924 15:25:39.556599 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.556639 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.556706 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.556892 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.556898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.557282 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:39.557341 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.557344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.558466 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.558467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.559024 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.559026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.559516 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.559517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.559602 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.560025 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:39.560028 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:39.561258 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:39.562899 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.562905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.562907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.563099 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.563333 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.563334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.563334 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.563399 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.563676 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:39.563683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:39.563739 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.563835 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.563949 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":85,"threads_started":1}
I20260924 15:25:39.564042 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.556617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.4.78.65:43829)
0924 15:25:39.556748 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.556752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.556761 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:25:39.556792 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.556793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.556805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.556872 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:39.557429 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.558396 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.559126 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.559423 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.559499 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.561345 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:39.561352 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:39.561353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:39.561360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:39.562798 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.563195 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.563198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.563198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.563262 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.563465 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.563466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.563466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.563598 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:39.563684 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.563873 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.563958 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260924 15:25:39.564482 18874 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fe8d6803d74686b80e7a23e9318085 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:39 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:45368
I20260924 15:25:39.565968 18949 tablet_replica.cc:330] stopping tablet replica
I20260924 15:25:39.566088 18949 raft_consensus.cc:2229] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:25:39.566184 18949 raft_consensus.cc:2258] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:39.566458 18949 ts_tablet_manager.cc:1905] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:39.568393 18949 ts_tablet_manager.cc:1918] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:39.568488 18949 log.cc:1196] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/65fe8d6803d74686b80e7a23e9318085
I20260924 15:25:39.568724 18949 ts_tablet_manager.cc:1939] T 65fe8d6803d74686b80e7a23e9318085 P 47d9a98618b14d648b4b363eb4d27b63: Deleting consensus metadata
I20260924 15:25:39.569197 18739 catalog_manager.cc:4841] TS 47d9a98618b14d648b4b363eb4d27b63 (127.4.78.65:43829): tablet 65fe8d6803d74686b80e7a23e9318085 (table test-table [id=283b2fa90e124d0196de6aa65308e1a3]) successfully deleted
W20260924 15:25:39.631234 18946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:39.631426 18946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:39.631449 18946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:39.632632 18946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:39.632683 18946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:39.632709 18946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:25:39.634265 18946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:34287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=36171
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:39.634509 18946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:39.634711 18946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:39.639842 18954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:39.639854 18957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:39.639868 18955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:39.640079 18946 server_base.cc:1034] running on GCE node
I20260924 15:25:39.640271 18946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:39.640527 18946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:39.641692 18946 hybrid_clock.cc:648] HybridClock initialized: now 1790263539641678 us; error 49 us; skew 500 ppm
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:39.644219 18946 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:25:39.644629 18946 webserver.cc:460] Webserver started at http://127.4.78.66:36171/ using document root <none> and password file <none>
I20260924 15:25:39.644873 18946 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:39.644956 18946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:39.646489 18946 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:25:39.647225 18964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.647385 18946 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:39.647471 18946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "7bc63a4e79a4434293936c0a80a68317"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "ec95fae3d92b97fcef267c30d2eb7800"
server_key_iv: "ae084687825de3aa74f29ef857fa6a01"
server_key_version: "encryptionkey@0"
I20260924 15:25:39.647768 18946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:39.665449 18946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:39.667325 18946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:39.667429 18946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:39.667603 18946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:39.667838 18946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:39.667868 18946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.667889 18946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:39.667902 18946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.673425 18946 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:34287
I20260924 15:25:39.673499 19077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:34287 every 8 connection(s)
I20260924 15:25:39.673904 18946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:39.682775 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.674667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:52465)
0924 15:25:39.674859 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.674898 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.675356 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.675579 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.675582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.676115 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:39.676298 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.676302 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.676926 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.676927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.677714 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.677716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.678096 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.678097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.678196 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.678706 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:39.678722 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:39.680052 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:39.681565 (+ 1513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.681571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.681572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.681781 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.681997 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.681997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.681997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.682091 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.682302 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:39.682306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:39.682411 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.682492 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.682618 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
I20260924 15:25:39.682787 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.674695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:52465 (local address 127.4.78.126:45845)
0924 15:25:39.674761 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.674763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.674983 (+ 220us) server_negotiation.cc:407] Connection header received
0924 15:25:39.675435 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.675436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.675450 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.675508 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:39.676370 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.676852 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.677792 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.678033 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.678080 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.680137 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:39.680138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:39.680138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:39.680145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:39.681492 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.681883 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.681886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.681887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.681940 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.682148 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.682148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.682148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.682247 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:39.682296 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.682554 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.682675 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:25:39.682812 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 18946
I20260924 15:25:39.683082 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:45849
--local_ip_for_outbound_sockets=127.4.78.67
--tserver_master_addrs=127.4.78.126:45845
--webserver_port=33803
--webserver_interface=127.4.78.67
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:25:39.683606 19078 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45845
I20260924 15:25:39.683687 19078 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:39.683867 19078 heartbeater.cc:511] Master 127.4.78.126:45845 requested a full tablet report, sending...
I20260924 15:25:39.684285 18752 ts_manager.cc:194] Registered new tserver with Master: 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
I20260924 15:25:39.684796 18752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:52465
W20260924 15:25:39.764925 19081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:39.765141 19081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:39.765275 19081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:39.766499 19081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:39.766564 19081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:39.766613 19081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:25:39.768067 19081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:45849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=33803
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:39.768334 19081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:39.768575 19081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:39.773433 19089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:39.773427 19087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:39.773448 19086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:39.773737 19081 server_base.cc:1034] running on GCE node
I20260924 15:25:39.773957 19081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:39.774240 19081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:39.775380 19081 hybrid_clock.cc:648] HybridClock initialized: now 1790263539775371 us; error 27 us; skew 500 ppm
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:39.777729 19081 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:25:39.778258 19081 webserver.cc:460] Webserver started at http://127.4.78.67:33803/ using document root <none> and password file <none>
I20260924 15:25:39.778496 19081 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:39.778576 19081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:39.779883 19081 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:25:39.780588 19096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.780751 19081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:25:39.780851 19081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "6f562ebd694a4a9f9bba64fc9040eb4c"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "43a0a7f021b8e1bc977940e0bd16471b"
server_key_iv: "130b665e15434b384cd5b7346a27b968"
server_key_version: "encryptionkey@0"
I20260924 15:25:39.781221 19081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:39.806195 19081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:39.808107 19081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:39.808277 19081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:39.808526 19081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:39.808781 19081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:39.808810 19081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.808833 19081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:39.808845 19081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:39.814875 19081 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:45849
I20260924 15:25:39.814917 19209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:45849 every 8 connection(s)
I20260924 15:25:39.815301 19081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:25:39.818607 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 19081
Sep 24 15:25:39 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263539, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:39.824939 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.815984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:60747 (local address 127.4.78.126:45845)
0924 15:25:39.816029 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.816031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.816268 (+ 237us) server_negotiation.cc:407] Connection header received
0924 15:25:39.816720 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.816722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.816731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.816783 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:39.817646 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.818144 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.819580 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.819762 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.819807 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.822188 (+ 2381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:39.822190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:39.822190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:39.822197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:39.823529 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.823940 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.823941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.823942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.824001 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.824298 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.824299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.824300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.824413 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:39.824490 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.824770 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.824850 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:25:39.825124 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.815989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:60747)
0924 15:25:39.816139 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.816177 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.816635 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.816835 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.816838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.817381 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:39.817572 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.817576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.818245 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.818246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.819499 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.819502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.819840 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.819841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.819990 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.820482 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:39.820490 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:39.822087 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:39.823624 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.823630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.823632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.823861 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.824070 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:39.824071 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:39.824071 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:39.824224 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:39.824481 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:39.824488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:39.824626 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.824734 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.824887 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260924 15:25:39.825588 19210 heartbeater.cc:347] Connected to a master server at 127.4.78.126:45845
I20260924 15:25:39.825692 19210 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:39.825904 19210 heartbeater.cc:511] Master 127.4.78.126:45845 requested a full tablet report, sending...
I20260924 15:25:39.825948 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.819019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.4.78.126:45845)
0924 15:25:39.819110 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.819112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.819152 (+ 40us) server_negotiation.cc:407] Connection header received
0924 15:25:39.819233 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.819233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.819242 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.819300 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:39.819977 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.820572 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.821497 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.821857 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.821940 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.823239 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:39.823240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:39.823240 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:39.823248 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:39.824620 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.824962 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.824963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.824963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.825026 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:39.825319 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:39.825320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:39.825320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:39.825445 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:39.825634 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.825716 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.825856 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74,"thread_start_us":46,"threads_started":1}
I20260924 15:25:39.826313 18752 ts_manager.cc:194] Registered new tserver with Master: 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849)
I20260924 15:25:39.826696 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:25:39.826768 18752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:60747
I20260924 15:25:39.829272 19214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.827017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44952 (local address 127.4.78.126:45845)
0924 15:25:39.827050 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.827051 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.827112 (+ 61us) server_negotiation.cc:407] Connection header received
0924 15:25:39.827187 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.827188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.827217 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.827294 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:25:39.827376 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.827885 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.828578 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.828752 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.828807 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.828933 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:25:39.829012 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:25:39.829126 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.829176 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.829199 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:25:39.829566 18752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:25:39.831171 18752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:25:39.833244 18874 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef2d2c557f94db7a994a70fa596285d (DEFAULT_TABLE table=test-table [id=e2b73d3e5bc8417ca097c99bec183c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:39.833417 18874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef2d2c557f94db7a994a70fa596285d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:39.834831 18832 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Bootstrap starting.
I20260924 15:25:39.835170 18832 tablet_bootstrap.cc:654] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:39.835692 18832 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: No bootstrap required, opened a new log
I20260924 15:25:39.835752 18832 ts_tablet_manager.cc:1397] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:39.836110 18832 raft_consensus.cc:348] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:39.836190 18832 raft_consensus.cc:374] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:39.836236 18832 raft_consensus.cc:729] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d9a98618b14d648b4b363eb4d27b63, State: Initialized, Role: FOLLOWER
I20260924 15:25:39.836290 18832 consensus_queue.cc:260] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:39.836378 18832 ts_tablet_manager.cc:1428] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:25:39.836957 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.832922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:45849 (local address 127.0.0.1:56254)
0924 15:25:39.833131 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.833141 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.833189 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.833381 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.833382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.833384 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.833422 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.833424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.834547 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.834548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.835676 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.835679 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.836645 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.836647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.836725 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.836795 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.836847 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.836891 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":43}
I20260924 15:25:39.837181 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.832957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.4.78.67:45849)
0924 15:25:39.833148 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.833153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.833179 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:25:39.833259 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.833261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.833274 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.833361 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.833473 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.834438 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.835773 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.836532 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.836618 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.836677 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.837058 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.837097 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260924 15:25:39.837790 19144 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef2d2c557f94db7a994a70fa596285d (DEFAULT_TABLE table=test-table [id=e2b73d3e5bc8417ca097c99bec183c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:39.838166 19144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef2d2c557f94db7a994a70fa596285d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:39.838766 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.833006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:34287 (local address 127.0.0.1:36884)
0924 15:25:39.833147 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.833165 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.833213 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.835246 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.835247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.835250 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.835298 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.835300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.836623 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.836624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.837634 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.837637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.838585 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.838586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.838633 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.838636 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:25:39.838692 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:25:39.838722 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 15:25:39.838830 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.834902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.4.78.66:34287)
0924 15:25:39.835074 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.835079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.835089 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:25:39.835127 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.835129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.835143 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.835221 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.835376 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.836510 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.837734 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.838503 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.838583 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:39.838640 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:39.838705 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:25:39.838746 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260924 15:25:39.839437 19012 tablet_service.cc:1467] Processing CreateTablet for tablet 0ef2d2c557f94db7a994a70fa596285d (DEFAULT_TABLE table=test-table [id=e2b73d3e5bc8417ca097c99bec183c91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:39.839773 19012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ef2d2c557f94db7a994a70fa596285d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:39.841789 19220 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: Bootstrap starting.
I20260924 15:25:39.842423 19220 tablet_bootstrap.cc:654] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:39.842887 19220 log.cc:824] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:39.843014 19221 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Bootstrap starting.
I20260924 15:25:39.843613 19221 tablet_bootstrap.cc:654] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:39.843739 19220 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: No bootstrap required, opened a new log
I20260924 15:25:39.843828 19220 ts_tablet_manager.cc:1397] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:25:39.844133 19221 log.cc:824] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:39.844832 19221 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: No bootstrap required, opened a new log
I20260924 15:25:39.844883 19221 ts_tablet_manager.cc:1397] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:25:39.845543 19220 raft_consensus.cc:348] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:39.845753 19220 raft_consensus.cc:374] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:39.845827 19220 raft_consensus.cc:729] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f562ebd694a4a9f9bba64fc9040eb4c, State: Initialized, Role: FOLLOWER
I20260924 15:25:39.845956 19220 consensus_queue.cc:260] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:39.846230 19220 ts_tablet_manager.cc:1428] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:25:39.846242 19210 heartbeater.cc:503] Master 127.4.78.126:45845 was elected leader, sending a full tablet report...
I20260924 15:25:39.846839 19221 raft_consensus.cc:348] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:39.847004 19221 raft_consensus.cc:374] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:39.847039 19221 raft_consensus.cc:729] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bc63a4e79a4434293936c0a80a68317, State: Initialized, Role: FOLLOWER
I20260924 15:25:39.847146 19221 consensus_queue.cc:260] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:39.847397 19221 ts_tablet_manager.cc:1428] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:25:39.847445 19078 heartbeater.cc:503] Master 127.4.78.126:45845 was elected leader, sending a full tablet report...
W20260924 15:25:39.924479 19079 tablet.cc:2369] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:25:39.995862 18945 raft_consensus.cc:482] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:39.996187 18945 raft_consensus.cc:504] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:39.996681 18945 leader_election.cc:288] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849), 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
I20260924 15:25:40.000772 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.997019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:54077 (local address 127.4.78.67:45849)
0924 15:25:39.997073 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.997075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.997090 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:25:39.997124 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.997125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.997138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.997209 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.997402 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.998251 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.999783 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:40.000361 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:40.000440 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:40.000450 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:40.000655 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 15:25:40.000692 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:25:40.000762 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.997120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:35011 (local address 127.4.78.66:34287)
0924 15:25:39.997172 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:39.997174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:39.997245 (+ 71us) server_negotiation.cc:407] Connection header received
0924 15:25:39.997295 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:39.997296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:39.997306 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:39.997348 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:39.997531 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.998271 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.999780 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:40.000376 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:40.000440 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:40.000450 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:40.000655 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 15:25:40.000692 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:25:40.000798 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.996894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:45849 (local address 127.4.78.65:54077)
0924 15:25:39.996964 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.996978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.997054 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.997218 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.997219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.997222 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.997305 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.997308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.998368 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.998369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.999677 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.999680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:40.000458 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:40.000459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:40.000557 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:40.000570 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:25:40.000633 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:25:40.000680 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":79}
I20260924 15:25:40.000806 19226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:39.997002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:34287 (local address 127.4.78.65:35011)
0924 15:25:39.997167 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:39.997176 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:39.997224 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:39.997376 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:39.997377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:39.997379 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:39.997450 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.997451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:39.998368 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:39.998369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:39.999677 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:39.999680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:40.000459 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:40.000460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:40.000543 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:40.000570 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 15:25:40.000640 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:25:40.000678 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":19,"thread_start_us":97,"threads_started":1}
I20260924 15:25:40.001186 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef2d2c557f94db7a994a70fa596285d" candidate_uuid: "47d9a98618b14d648b4b363eb4d27b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" is_pre_election: true
I20260924 15:25:40.001255 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef2d2c557f94db7a994a70fa596285d" candidate_uuid: "47d9a98618b14d648b4b363eb4d27b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc63a4e79a4434293936c0a80a68317" is_pre_election: true
I20260924 15:25:40.001376 19032 raft_consensus.cc:2454] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d9a98618b14d648b4b363eb4d27b63 in term 0.
I20260924 15:25:40.001376 19164 raft_consensus.cc:2454] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47d9a98618b14d648b4b363eb4d27b63 in term 0.
I20260924 15:25:40.001597 18826 leader_election.cc:302] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d9a98618b14d648b4b363eb4d27b63, 6f562ebd694a4a9f9bba64fc9040eb4c; no voters:
I20260924 15:25:40.001724 18945 raft_consensus.cc:2781] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:25:40.001806 18945 raft_consensus.cc:482] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:25:40.001868 18945 raft_consensus.cc:3037] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:40.002616 18945 raft_consensus.cc:504] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:40.002761 18945 leader_election.cc:288] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 1 election: Requested vote from peers 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849), 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
I20260924 15:25:40.002954 19164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef2d2c557f94db7a994a70fa596285d" candidate_uuid: "47d9a98618b14d648b4b363eb4d27b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c"
I20260924 15:25:40.002979 19032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ef2d2c557f94db7a994a70fa596285d" candidate_uuid: "47d9a98618b14d648b4b363eb4d27b63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7bc63a4e79a4434293936c0a80a68317"
I20260924 15:25:40.003041 19164 raft_consensus.cc:3037] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:40.003044 19032 raft_consensus.cc:3037] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:40.003938 19164 raft_consensus.cc:2454] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d9a98618b14d648b4b363eb4d27b63 in term 1.
I20260924 15:25:40.003938 19032 raft_consensus.cc:2454] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47d9a98618b14d648b4b363eb4d27b63 in term 1.
I20260924 15:25:40.004132 18826 leader_election.cc:302] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47d9a98618b14d648b4b363eb4d27b63, 7bc63a4e79a4434293936c0a80a68317; no voters:
I20260924 15:25:40.004233 18945 raft_consensus.cc:2781] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:40.004338 18945 raft_consensus.cc:686] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 LEADER]: Becoming Leader. State: Replica: 47d9a98618b14d648b4b363eb4d27b63, State: Running, Role: LEADER
I20260924 15:25:40.004410 18945 consensus_queue.cc:237] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:40.005154 18751 catalog_manager.cc:5495] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47d9a98618b14d648b4b363eb4d27b63 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "47d9a98618b14d648b4b363eb4d27b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } health_report { overall_health: HEALTHY } } }
I20260924 15:25:40.018898 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.016457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.4.78.65:43829)
0924 15:25:40.016521 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:40.016523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:40.016533 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:25:40.016606 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:40.016607 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:40.016634 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:40.016710 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:25:40.016844 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:40.017322 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:40.018108 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:40.018324 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:40.018412 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:40.018582 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:25:40.018680 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:25:40.018768 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:40.018791 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 15:25:40.018812 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:25:40.020462 19032 raft_consensus.cc:1261] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 1 FOLLOWER]: Refusing update from remote peer 47d9a98618b14d648b4b363eb4d27b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:40.020493 19164 raft_consensus.cc:1261] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 FOLLOWER]: Refusing update from remote peer 47d9a98618b14d648b4b363eb4d27b63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:40.020835 19227 consensus_queue.cc:1035] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:40.020963 18945 consensus_queue.cc:1035] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:40.023300 19230 mvcc.cc:205] Tried to move back new op lower bound from 7332919459920986112 to 7332919459858698240. Current Snapshot: MvccSnapshot[applied={T|T < 7332919459920986112}]
I20260924 15:25:40.023413 19231 mvcc.cc:205] Tried to move back new op lower bound from 7332919459920986112 to 7332919459858698240. Current Snapshot: MvccSnapshot[applied={T|T < 7332919459920986112}]
I20260924 15:25:40.023813 19232 mvcc.cc:205] Tried to move back new op lower bound from 7332919459920986112 to 7332919459858698240. Current Snapshot: MvccSnapshot[applied={T|T < 7332919459920986112}]
I20260924 15:25:40.025897 18752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:25:40.025990 18752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:25:40.026691 18752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Sending DeleteTablet for 3 replicas of tablet 0ef2d2c557f94db7a994a70fa596285d
I20260924 15:25:40.027004 19144 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef2d2c557f94db7a994a70fa596285d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:40 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:56254
I20260924 15:25:40.027058 19012 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ef2d2c557f94db7a994a70fa596285d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:40 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:36884
I20260924 15:25:40.027231 19240 tablet_replica.cc:330] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: stopping tablet replica
I20260924 15:25:40.027294 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 18790
I20260924 15:25:40.027340 19240 raft_consensus.cc:2229] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:25:40.027458 19240 raft_consensus.cc:2258] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:40.028003 19240 ts_tablet_manager.cc:1905] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:40.030004 19240 ts_tablet_manager.cc:1918] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:40.030081 19240 log.cc:1196] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/0ef2d2c557f94db7a994a70fa596285d
I20260924 15:25:40.030380 19240 ts_tablet_manager.cc:1939] T 0ef2d2c557f94db7a994a70fa596285d P 6f562ebd694a4a9f9bba64fc9040eb4c: Deleting consensus metadata
I20260924 15:25:40.030952 18736 catalog_manager.cc:4841] TS 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849): tablet 0ef2d2c557f94db7a994a70fa596285d (table test-table [id=e2b73d3e5bc8417ca097c99bec183c91]) successfully deleted
I20260924 15:25:40.033452 19241 tablet_replica.cc:330] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: stopping tablet replica
I20260924 15:25:40.033574 19241 raft_consensus.cc:2229] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:25:40.033684 19241 raft_consensus.cc:2258] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:40.034090 19241 ts_tablet_manager.cc:1905] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:40.036026 19241 ts_tablet_manager.cc:1918] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:40.036115 19241 log.cc:1196] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/0ef2d2c557f94db7a994a70fa596285d
I20260924 15:25:40.036397 19241 ts_tablet_manager.cc:1939] T 0ef2d2c557f94db7a994a70fa596285d P 7bc63a4e79a4434293936c0a80a68317: Deleting consensus metadata
W20260924 15:25:40.036743 18739 connection.cc:535] client connection to 127.4.78.65:43829 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:25:40.036794 18736 catalog_manager.cc:4841] TS 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287): tablet 0ef2d2c557f94db7a994a70fa596285d (table test-table [id=e2b73d3e5bc8417ca097c99bec183c91]) successfully deleted
W20260924 15:25:40.036854 18739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:25:40.037273 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 18946
I20260924 15:25:40.037417 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.037265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:43829 (local address 127.0.0.1:45396)
0924 15:25:40.037325 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.037381 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:43829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 15:25:40.037957 18739 catalog_manager.cc:4584] TS 47d9a98618b14d648b4b363eb4d27b63 (127.4.78.65:43829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ef2d2c557f94db7a994a70fa596285d: Network error: Client connection negotiation failed: client connection to 127.4.78.65:43829: connect: Connection refused (error 111)
I20260924 15:25:40.042147 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 19081
I20260924 15:25:40.046707 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 18719
I20260924 15:25:40.052021 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:45845
--webserver_interface=127.4.78.126
--webserver_port=45279
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:45845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:40.131709 19243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:40.131898 19243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:40.131920 19243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:40.133167 19243 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:40.133203 19243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:40.133217 19243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:40.133230 19243 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:40.133242 19243 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:40.134943 19243 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:45845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:45845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=45279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:40.135172 19243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:40.135390 19243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:40.140278 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:40.140267 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:40.140318 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:40.140676 19243 server_base.cc:1034] running on GCE node
I20260924 15:25:40.140918 19243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:40.141202 19243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:40.142336 19243 hybrid_clock.cc:648] HybridClock initialized: now 1790263540142314 us; error 38 us; skew 500 ppm
Sep 24 15:25:40 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263540, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:40.144726 19243 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:25:40.145180 19243 webserver.cc:460] Webserver started at http://127.4.78.126:45279/ using document root <none> and password file <none>
I20260924 15:25:40.145412 19243 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:40.145483 19243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:40.146826 19243 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:40.147480 19258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:40.147644 19243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:25:40.147746 19243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e4199575c4ee4f9d8ceb31f90a73be86"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "217aad2d5ab0fab0a548f53981e0dfe0"
server_key_iv: "701553e2b887c159e428a4dfbd112cad"
server_key_version: "encryptionkey@0"
I20260924 15:25:40.148078 19243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:40.172807 19243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:40.175148 19243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:40.175320 19243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:40.179479 19243 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:45845
I20260924 15:25:40.179518 19310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:45845 every 8 connection(s)
I20260924 15:25:40.179924 19243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:25:40.181492 19311 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:25:40.182216 19311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Bootstrap starting.
I20260924 15:25:40.187098 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 19243
I20260924 15:25:40.187426 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:43829
--local_ip_for_outbound_sockets=127.4.78.65
--tserver_master_addrs=127.4.78.126:45845
--webserver_port=41213
--webserver_interface=127.4.78.65
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:25:40.190809 19311 log.cc:824] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:40.193034 19311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:25:40.193343 19311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86: Bootstrap complete.
I20260924 15:25:40.195591 19311 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:40.195802 19311 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4199575c4ee4f9d8ceb31f90a73be86, State: Initialized, Role: FOLLOWER
I20260924 15:25:40.195968 19311 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:40.196050 19311 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:25:40.196105 19311 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:25:40.196185 19311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 2 FOLLOWER]: Advancing to term 3
I20260924 15:25:40.197190 19311 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:40.197309 19311 leader_election.cc:302] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4199575c4ee4f9d8ceb31f90a73be86; no voters:
I20260924 15:25:40.197504 19311 leader_election.cc:288] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 15:25:40.197540 19315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 15:25:40.197642 19315 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [term 3 LEADER]: Becoming Leader. State: Replica: e4199575c4ee4f9d8ceb31f90a73be86, State: Running, Role: LEADER
I20260924 15:25:40.197746 19315 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } }
I20260924 15:25:40.197772 19311 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:25:40.198199 19316 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } } }
I20260924 15:25:40.198299 19316 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:40.198714 19321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:25:40.199131 19317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4199575c4ee4f9d8ceb31f90a73be86. Latest consensus state: current_term: 3 leader_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4199575c4ee4f9d8ceb31f90a73be86" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 45845 } } }
I20260924 15:25:40.199263 19317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4199575c4ee4f9d8ceb31f90a73be86 [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:40.199468 19321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:25:40.199564 19321 catalog_manager.cc:1259] Loaded cluster ID: 6e307147fafe4456a7675f0395260277
I20260924 15:25:40.199589 19321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 15:25:40.199820 19321 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff7032959ad google::LogMessage::Fail() at ??:0
@ 0x7ff70329888a google::LogMessage::SendToLog() at ??:0
@ 0x7ff7032954de google::LogMessage::Flush() at ??:0
@ 0x7ff703296a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff70b1b60f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7ff70b1bf8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff7051630d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff705158954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff705fbc6db start_thread at ??:0
@ 0x7ff701d9361f clone at ??:0
W20260924 15:25:40.305938 19313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:40.306207 19313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:40.306245 19313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:40.308148 19313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:40.308209 19313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:40.308248 19313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:25:40.310504 19313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:43829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=41213
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:40.310761 19313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:40.311020 19313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:40.316291 19347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:40.316264 19348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:40.316278 19350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:40.316476 19313 server_base.cc:1034] running on GCE node
I20260924 15:25:40.316737 19313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:40.316972 19313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:40.318138 19313 hybrid_clock.cc:648] HybridClock initialized: now 1790263540318120 us; error 30 us; skew 500 ppm
Sep 24 15:25:40 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263540, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:40.321074 19313 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:25:40.321527 19313 webserver.cc:460] Webserver started at http://127.4.78.65:41213/ using document root <none> and password file <none>
I20260924 15:25:40.321754 19313 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:40.321847 19313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:40.323194 19313 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:25:40.323897 19357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:40.324075 19313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:25:40.324165 19313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "47d9a98618b14d648b4b363eb4d27b63"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "586ce03806528c826d67cc9b5d08573f"
server_key_iv: "28d099991830cb6a8231dcf8454f5eb2"
server_key_version: "encryptionkey@0"
I20260924 15:25:40.324501 19313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:40.335983 19313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:40.338330 19313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:40.338479 19313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:40.338702 19313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:40.339134 19364 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:25:40.341049 19313 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:25:40.341183 19313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:25:40.341230 19313 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:25:40.341974 19313 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:25:40.342024 19313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:40.342099 19364 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Bootstrap starting.
I20260924 15:25:40.349190 19313 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:43829
I20260924 15:25:40.349270 19471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:43829 every 8 connection(s)
I20260924 15:25:40.349597 19313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:25:40.353080 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 19313
I20260924 15:25:40.353399 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:34287
--local_ip_for_outbound_sockets=127.4.78.66
--tserver_master_addrs=127.4.78.126:45845
--webserver_port=36171
--webserver_interface=127.4.78.66
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:25:40.353425 19364 log.cc:824] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:40.356046 19364 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:25:40.356359 19364 tablet_bootstrap.cc:492] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Bootstrap complete.
I20260924 15:25:40.357583 19364 ts_tablet_manager.cc:1397] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260924 15:25:40.359009 19364 raft_consensus.cc:348] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:40.359341 19364 raft_consensus.cc:729] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47d9a98618b14d648b4b363eb4d27b63, State: Initialized, Role: FOLLOWER
I20260924 15:25:40.359493 19364 consensus_queue.cc:260] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:40.359772 19364 ts_tablet_manager.cc:1428] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260924 15:25:40.413410 19474 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:25:40.350327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:37095)
0924 15:25:40.350519 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.350555 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:40.350981 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:40.413201 (+ 62220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
W20260924 15:25:40.413721 19472 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:45845 (0 consecutive failures): Network error: Failed to ping master at 127.4.78.126:45845: Client connection negotiation failed: client connection to 127.4.78.126:45845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:25:40.414153 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.414018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:47931)
0924 15:25:40.414075 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.414126 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:25:40.414510 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.414405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:41395)
0924 15:25:40.414453 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.414477 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:25:40.414665 19472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:25:40.436849 19476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:40.437037 19476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:40.437060 19476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:40.438302 19476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:40.438344 19476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:40.438371 19476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:25:40.439792 19476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:34287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=36171
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:40.439967 19476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:40.440193 19476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:40.444963 19483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:40.444964 19485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:40.444962 19482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:40.445426 19476 server_base.cc:1034] running on GCE node
I20260924 15:25:40.445662 19476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:40.445935 19476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:40.447105 19476 hybrid_clock.cc:648] HybridClock initialized: now 1790263540447094 us; error 27 us; skew 500 ppm
Sep 24 15:25:40 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263540, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:40.449640 19476 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:25:40.450169 19476 webserver.cc:460] Webserver started at http://127.4.78.66:36171/ using document root <none> and password file <none>
I20260924 15:25:40.450393 19476 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:40.450438 19476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:40.451742 19476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:40.452327 19492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:40.452503 19476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:25:40.452620 19476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "7bc63a4e79a4434293936c0a80a68317"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "ec95fae3d92b97fcef267c30d2eb7800"
server_key_iv: "ae084687825de3aa74f29ef857fa6a01"
server_key_version: "encryptionkey@0"
I20260924 15:25:40.452955 19476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:40.467043 19476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:40.469274 19476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:40.469455 19476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:40.469700 19476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:40.470037 19476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:40.470093 19476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:40.470129 19476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:40.470155 19476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:40.475904 19476 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:34287
I20260924 15:25:40.475953 19605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:34287 every 8 connection(s)
I20260924 15:25:40.476272 19476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:25:40.477205 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.476854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:53477)
0924 15:25:40.476999 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.477064 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
W20260924 15:25:40.477612 19606 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:45845 (0 consecutive failures): Network error: Failed to ping master at 127.4.78.126:45845: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
I20260924 15:25:40.477927 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.477800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:33185)
0924 15:25:40.477886 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.477911 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 15:25:40.478168 19608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.478115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:48071)
0924 15:25:40.478136 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.478155 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 15:25:40.478271 19606 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:25:40.479043 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 19476
I20260924 15:25:40.479308 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:45849
--local_ip_for_outbound_sockets=127.4.78.67
--tserver_master_addrs=127.4.78.126:45845
--webserver_port=33803
--webserver_interface=127.4.78.67
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:40.562014 19609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:40.562196 19609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:40.562217 19609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:40.563282 19609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:40.563314 19609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:40.563338 19609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:25:40.564666 19609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:45849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=33803
--tserver_master_addrs=127.4.78.126:45845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:40.564845 19609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:40.565002 19609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:40.569855 19617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:40.569902 19614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:40.569861 19615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:40.570087 19609 server_base.cc:1034] running on GCE node
I20260924 15:25:40.570307 19609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:40.570549 19609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:40.571723 19609 hybrid_clock.cc:648] HybridClock initialized: now 1790263540571723 us; error 39 us; skew 500 ppm
Sep 24 15:25:40 dist-test-slave-l766 krb5kdc[18168](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263540, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:40.574122 19609 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:25:40.574533 19609 webserver.cc:460] Webserver started at http://127.4.78.67:33803/ using document root <none> and password file <none>
I20260924 15:25:40.574762 19609 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:40.574807 19609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:40.576089 19609 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:40.576680 19624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:40.576867 19609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:25:40.576951 19609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "6f562ebd694a4a9f9bba64fc9040eb4c"
format_stamp: "Formatted at 2026-09-24 15:25:38 on dist-test-slave-l766"
server_key: "43a0a7f021b8e1bc977940e0bd16471b"
server_key_iv: "130b665e15434b384cd5b7346a27b968"
server_key_version: "encryptionkey@0"
I20260924 15:25:40.577277 19609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:40.592470 19609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:40.594379 19609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:40.594486 19609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:40.594662 19609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:40.594954 19609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:40.594985 19609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:40.595007 19609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:40.595053 19609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:40.600919 19609 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:45849
I20260924 15:25:40.600996 19737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:45849 every 8 connection(s)
I20260924 15:25:40.601334 19609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:25:40.602427 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.602054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:55425)
0924 15:25:40.602210 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.602288 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260924 15:25:40.602882 19738 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:45845 (0 consecutive failures): Network error: Failed to ping master at 127.4.78.126:45845: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
I20260924 15:25:40.603191 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.603102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:39799)
0924 15:25:40.603148 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.603174 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:25:40.603490 19740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:40.603416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:37937)
0924 15:25:40.603453 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:40.603475 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 15:25:40.603632 19738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:25:40.604094 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 19609
I20260924 15:25:41.415252 19742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:41.415022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:48891)
0924 15:25:41.415145 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:41.415190 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260924 15:25:41.478781 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:41.478565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:40647)
0924 15:25:41.478697 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:41.478733 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260924 15:25:41.604132 19744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:41.603942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:56765)
0924 15:25:41.604052 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:41.604084 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":55,"threads_started":1}
I20260924 15:25:41.970038 19745 raft_consensus.cc:482] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:41.970148 19745 raft_consensus.cc:504] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:41.970453 19745 leader_election.cc:288] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849), 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
Sep 24 15:25:41 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263540, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.67@KRBTEST.COM
Sep 24 15:25:41 dist-test-slave-l766 krb5kdc[18168](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263540, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.66@KRBTEST.COM
I20260924 15:25:41.980762 19748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:41.970778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:34287 (local address 127.4.78.65:60203)
0924 15:25:41.970875 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:41.970884 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:41.970923 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:41.971748 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:41.971750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:41.972087 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:41.972265 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:41.972270 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:41.973381 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:41.973382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:41.974175 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:41.974177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:41.974755 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:41.974764 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:41.974874 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:41.975519 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:41.975524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:41.977425 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:41.979441 (+ 2016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:41.979442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:41.979443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:41.979617 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:41.979879 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:41.979880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:41.979880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:41.979936 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:41.980385 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:41.980386 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:41.980424 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:25:41.980490 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:25:41.980643 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260924 15:25:41.980778 19747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:41.970730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:44719 (local address 127.4.78.67:45849)
0924 15:25:41.970946 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:41.970957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:41.970987 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:25:41.971428 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:41.971430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:41.971471 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:41.971588 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:41.972397 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:41.973228 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:41.974306 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:41.974687 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:41.974811 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:41.977093 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:41.977097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:41.977098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:41.977108 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:41.978878 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:41.979316 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:41.979321 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:41.979321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:41.979435 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:41.979746 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:41.979747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:41.979747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:41.980116 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:41.980209 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:41.980539 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 15:25:41.980672 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260924 15:25:41.980772 19749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:41.970923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:60203 (local address 127.4.78.66:34287)
0924 15:25:41.971067 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:41.971076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:41.971102 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:25:41.971537 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:41.971540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:41.971615 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:41.971755 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:41.972382 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:41.973257 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:41.974256 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:41.974666 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:41.974773 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:41.977519 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:41.977522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:41.977522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:41.977533 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:41.979367 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:41.979688 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:41.979690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:41.979691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:41.979809 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:41.980021 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:41.980022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:41.980022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:41.980316 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:41.980373 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:41.980556 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 15:25:41.980688 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":57,"threads_started":1}
I20260924 15:25:41.980892 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:41.970591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:45849 (local address 127.4.78.65:44719)
0924 15:25:41.970718 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:41.970729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:41.970786 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:41.971663 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:41.971676 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:41.972090 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:41.972264 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:41.972270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:41.973343 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:41.973344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:41.974179 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:41.974182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:41.974790 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:41.974792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:41.974873 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:41.975439 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:41.975446 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:41.976985 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:41.978977 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:41.978984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:41.978985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:41.979213 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:41.979536 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:41.979537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:41.979538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:41.979677 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:41.980195 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:41.980200 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:41.980316 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 15:25:41.980447 (+ 131us) client_negotiation.cc:240] Negotiation successful
0924 15:25:41.980686 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":110,"thread_start_us":69,"threads_started":1}
W20260924 15:25:41.981716 19358 leader_election.cc:341] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849): Not found: Tablet not found: 0ef2d2c557f94db7a994a70fa596285d
W20260924 15:25:41.981814 19358 leader_election.cc:341] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287): Not found: Tablet not found: 0ef2d2c557f94db7a994a70fa596285d
I20260924 15:25:41.981879 19358 leader_election.cc:302] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47d9a98618b14d648b4b363eb4d27b63; no voters: 6f562ebd694a4a9f9bba64fc9040eb4c, 7bc63a4e79a4434293936c0a80a68317
I20260924 15:25:41.982036 19745 raft_consensus.cc:2726] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:25:42.415812 19746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:42.415690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:55183)
0924 15:25:42.415740 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:42.415771 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:25:42.479472 19750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:42.479241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:50671)
0924 15:25:42.479384 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:42.479423 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":74,"threads_started":1}
I20260924 15:25:42.604719 19751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:42.604522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:49517)
0924 15:25:42.604637 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:42.604681 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":52,"threads_started":1}
I20260924 15:25:43.416563 19752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:43.416326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:46069)
0924 15:25:43.416475 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:43.416513 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260924 15:25:43.480129 19753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:43.479909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:48193)
0924 15:25:43.480045 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:43.480083 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260924 15:25:43.605379 19754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:43.605158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:56217)
0924 15:25:43.605278 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:43.605318 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260924 15:25:43.893981 19755 raft_consensus.cc:482] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:43.894074 19755 raft_consensus.cc:504] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:43.894232 19755 leader_election.cc:288] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849), 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
W20260924 15:25:43.894690 19358 leader_election.cc:341] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287): Not found: Tablet not found: 0ef2d2c557f94db7a994a70fa596285d
W20260924 15:25:43.894757 19358 leader_election.cc:341] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849): Not found: Tablet not found: 0ef2d2c557f94db7a994a70fa596285d
I20260924 15:25:43.894810 19358 leader_election.cc:302] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47d9a98618b14d648b4b363eb4d27b63; no voters: 6f562ebd694a4a9f9bba64fc9040eb4c, 7bc63a4e79a4434293936c0a80a68317
I20260924 15:25:43.894886 19755 raft_consensus.cc:2726] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:25:44.417320 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:44.417103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:41653)
0924 15:25:44.417231 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:44.417272 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260924 15:25:44.480765 19757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:44.480582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:49853)
0924 15:25:44.480693 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:44.480726 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 15:25:44.606194 19758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:44.605929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:36609)
0924 15:25:44.606080 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:44.606136 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260924 15:25:45.411406 19759 raft_consensus.cc:482] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:45.411501 19759 raft_consensus.cc:504] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:45.411713 19759 leader_election.cc:288] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849), 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
W20260924 15:25:45.412190 19358 leader_election.cc:341] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849): Not found: Tablet not found: 0ef2d2c557f94db7a994a70fa596285d
W20260924 15:25:45.412283 19358 leader_election.cc:341] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287): Not found: Tablet not found: 0ef2d2c557f94db7a994a70fa596285d
I20260924 15:25:45.412320 19358 leader_election.cc:302] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47d9a98618b14d648b4b363eb4d27b63; no voters: 6f562ebd694a4a9f9bba64fc9040eb4c, 7bc63a4e79a4434293936c0a80a68317
I20260924 15:25:45.412416 19759 raft_consensus.cc:2726] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:25:45.418044 19760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:45.417861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:41713)
0924 15:25:45.417971 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:45.418010 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260924 15:25:45.481377 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:45.481177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:42361)
0924 15:25:45.481305 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:45.481340 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260924 15:25:45.606856 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:45.606641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:53397)
0924 15:25:45.606778 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:45.606815 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260924 15:25:46.418740 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:46.418501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:57073)
0924 15:25:46.418637 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:46.418681 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260924 15:25:46.482077 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:46.481884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:36905)
0924 15:25:46.482001 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:46.482036 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260924 15:25:46.607517 19765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:46.607320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:49729)
0924 15:25:46.607440 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:46.607475 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260924 15:25:47.419463 19766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:47.419251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:37091)
0924 15:25:47.419376 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:47.419414 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260924 15:25:47.482689 19767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:47.482470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:39341)
0924 15:25:47.482602 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:47.482644 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260924 15:25:47.608197 19768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:47.607980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:40405)
0924 15:25:47.608117 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:47.608156 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260924 15:25:48.420184 19769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:48.419973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:60315)
0924 15:25:48.420098 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:48.420136 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260924 15:25:48.483356 19770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:48.483144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:49423)
0924 15:25:48.483267 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:48.483303 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260924 15:25:48.608951 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:48.608646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:59707)
0924 15:25:48.608869 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:48.608908 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260924 15:25:49.420902 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:49.420653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:39819)
0924 15:25:49.420819 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:49.420854 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":61,"threads_started":1}
I20260924 15:25:49.484124 19773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:49.483911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:34323)
0924 15:25:49.484045 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:49.484080 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":61,"threads_started":1}
I20260924 15:25:49.609579 19774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:49.609390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:41211)
0924 15:25:49.609510 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:49.609540 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260924 15:25:50.325364 19775 raft_consensus.cc:482] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:50.325472 19775 raft_consensus.cc:504] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f562ebd694a4a9f9bba64fc9040eb4c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 45849 } } peers { permanent_uuid: "7bc63a4e79a4434293936c0a80a68317" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34287 } } peers { permanent_uuid: "47d9a98618b14d648b4b363eb4d27b63" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 43829 } }
I20260924 15:25:50.325675 19775 leader_election.cc:288] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849), 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287)
W20260924 15:25:50.326071 19358 leader_election.cc:341] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6f562ebd694a4a9f9bba64fc9040eb4c (127.4.78.67:45849): Not found: Tablet not found: 0ef2d2c557f94db7a994a70fa596285d
W20260924 15:25:50.326148 19358 leader_election.cc:341] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7bc63a4e79a4434293936c0a80a68317 (127.4.78.66:34287): Not found: Tablet not found: 0ef2d2c557f94db7a994a70fa596285d
I20260924 15:25:50.326165 19358 leader_election.cc:302] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47d9a98618b14d648b4b363eb4d27b63; no voters: 6f562ebd694a4a9f9bba64fc9040eb4c, 7bc63a4e79a4434293936c0a80a68317
I20260924 15:25:50.326233 19775 raft_consensus.cc:2726] T 0ef2d2c557f94db7a994a70fa596285d P 47d9a98618b14d648b4b363eb4d27b63 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:25:50.421674 19776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:50.421455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:54047)
0924 15:25:50.421587 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:50.421628 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260924 15:25:50.484812 19777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:50.484589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:38831)
0924 15:25:50.484723 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:50.484761 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260924 15:25:50.610262 19778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:50.610051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:53335)
0924 15:25:50.610167 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:50.610205 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260924 15:25:51.422536 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:51.422300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:50475)
0924 15:25:51.422445 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:51.422487 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260924 15:25:51.485491 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:51.485268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:35101)
0924 15:25:51.485408 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:51.485444 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260924 15:25:51.610966 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:51.610743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:48521)
0924 15:25:51.610888 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:51.610924 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260924 15:25:52.423276 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:52.423061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:44003)
0924 15:25:52.423185 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:52.423221 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260924 15:25:52.486238 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:52.486000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:51625)
0924 15:25:52.486151 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:52.486189 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260924 15:25:52.611688 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:52.611482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:58767)
0924 15:25:52.611608 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:52.611644 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260924 15:25:53.423987 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:53.423758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:45457)
0924 15:25:53.423900 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:53.423938 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260924 15:25:53.486965 19786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:53.486754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:54647)
0924 15:25:53.486879 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:53.486919 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260924 15:25:53.612418 19787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:53.612183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:46387)
0924 15:25:53.612333 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:53.612371 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260924 15:25:54.424844 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:54.424615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:44793)
0924 15:25:54.424743 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:54.424794 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260924 15:25:54.487620 19789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:54.487423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:37633)
0924 15:25:54.487541 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:54.487577 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260924 15:25:54.613231 19790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:54.612935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:54757)
0924 15:25:54.613110 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:54.613148 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260924 15:25:55.425676 19791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:55.425449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.65:47627)
0924 15:25:55.425598 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:55.425628 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":72,"threads_started":1}
I20260924 15:25:55.488412 19792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:55.488149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.66:58881)
0924 15:25:55.488329 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:55.488365 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":66,"threads_started":1}
I20260924 15:25:55.612643 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 19313
I20260924 15:25:55.614323 19793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:55.614091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:45845 (local address 127.4.78.67:53211)
0924 15:25:55.614242 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:55.614284 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:45845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":107,"threads_started":1}
I20260924 15:25:55.617507 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 19476
I20260924 15:25:55.621991 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 19609
2026-09-24T15:25:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18405 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:25:55 dist-test-slave-l766 krb5kdc[19796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:25:55 dist-test-slave-l766 krb5kdc[19796](info): set up 2 sockets
Sep 24 15:25:55 dist-test-slave-l766 krb5kdc[19796](info): commencing operation
krb5kdc: starting...
W20260924 15:25:56.664773 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:25:56 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:25:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:25:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:25:56.782915 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35221
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:35221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:56.863130 19809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:56.863332 19809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:56.863353 19809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:56.864495 19809 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:25:56.864534 19809 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:25:56.864547 19809 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:25:56.864559 19809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:25:56.864573 19809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:56.864624 19809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:56.864641 19809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:25:56.864665 19809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:25:56.866158 19809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:35221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:56.866412 19809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:56.866642 19809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:56.871644 19814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:56.871716 19815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:56.871644 19817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:56.871838 19809 server_base.cc:1034] running on GCE node
I20260924 15:25:56.872108 19809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:56.872401 19809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:56.873575 19809 hybrid_clock.cc:648] HybridClock initialized: now 1790263556873553 us; error 41 us; skew 500 ppm
Sep 24 15:25:56 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263556, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:56.876220 19809 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:25:56.876658 19809 webserver.cc:460] Webserver started at http://127.4.78.126:37487/ using document root <none> and password file <none>
I20260924 15:25:56.876910 19809 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:56.876956 19809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:56.877137 19809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:56.878284 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "ef602e27976248f3ba323b4996c9487f"
format_stamp: "Formatted at 2026-09-24 15:25:56 on dist-test-slave-l766"
server_key: "e7145ab9420572df70e138c14834f948"
server_key_iv: "733fcf2fa55a941f295948361b8ac19e"
server_key_version: "encryptionkey@0"
I20260924 15:25:56.878643 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "ef602e27976248f3ba323b4996c9487f"
format_stamp: "Formatted at 2026-09-24 15:25:56 on dist-test-slave-l766"
server_key: "e7145ab9420572df70e138c14834f948"
server_key_iv: "733fcf2fa55a941f295948361b8ac19e"
server_key_version: "encryptionkey@0"
I20260924 15:25:56.879966 19809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:56.880748 19824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:56.880954 19809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:56.881043 19809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "ef602e27976248f3ba323b4996c9487f"
format_stamp: "Formatted at 2026-09-24 15:25:56 on dist-test-slave-l766"
server_key: "e7145ab9420572df70e138c14834f948"
server_key_iv: "733fcf2fa55a941f295948361b8ac19e"
server_key_version: "encryptionkey@0"
I20260924 15:25:56.881116 19809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:56.898185 19809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:56.900081 19809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:56.900245 19809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:56.904368 19809 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:35221
I20260924 15:25:56.904431 19876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:35221 every 8 connection(s)
I20260924 15:25:56.904793 19809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:25:56.905658 19877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:56.907630 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 19809
I20260924 15:25:56.907730 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:25:56.907935 4409 external_mini_cluster.cc:1428] Setting key cd3e7093682f58f55acb12eb621ed362
I20260924 15:25:56.908326 19877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Bootstrap starting.
I20260924 15:25:56.908991 19877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:56.909298 19877 log.cc:824] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:56.910077 19877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: No bootstrap required, opened a new log
I20260924 15:25:56.912137 19877 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:25:56.912307 19877 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:56.912334 19877 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef602e27976248f3ba323b4996c9487f, State: Initialized, Role: FOLLOWER
I20260924 15:25:56.912483 19877 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:25:56.912544 19877 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:25:56.912565 19877 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:25:56.912629 19877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:56.913329 19877 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:25:56.913462 19877 leader_election.cc:302] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef602e27976248f3ba323b4996c9487f; no voters:
I20260924 15:25:56.913667 19877 leader_election.cc:288] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:25:56.913726 19882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:56.913916 19882 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 1 LEADER]: Becoming Leader. State: Replica: ef602e27976248f3ba323b4996c9487f, State: Running, Role: LEADER
I20260924 15:25:56.913993 19877 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: configured and running, proceeding with master startup.
Sep 24 15:25:56 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:56.914043 19882 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:25:56.915931 19884 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef602e27976248f3ba323b4996c9487f. Latest consensus state: current_term: 1 leader_uuid: "ef602e27976248f3ba323b4996c9487f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } } }
I20260924 15:25:56.916038 19884 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:56.915931 19883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef602e27976248f3ba323b4996c9487f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } } }
I20260924 15:25:56.916368 19897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:25:56.916400 19883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: This master's current role is: LEADER
I20260924 15:25:56.917001 19897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:25:56.918979 19897 catalog_manager.cc:1347] Generated new cluster ID: c14ada371e5643c098d826ef5c2939cc
I20260924 15:25:56.919044 19897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:25:56.919198 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:56.908482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.4.78.126:35221)
0924 15:25:56.908663 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:56.908671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:56.908705 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:25:56.909185 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:56.909202 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:56.909251 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:56.909386 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:56.910166 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:56.911066 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:56.912206 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:56.912809 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:56.912892 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:56.915704 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:56.915708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:56.915709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:56.915720 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:56.917518 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:56.917918 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:56.917921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:56.917922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:56.918021 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:56.918221 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:56.918222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:56.918222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:56.918528 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:56.918659 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:56.918777 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 15:25:56.918916 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
I20260924 15:25:56.930976 19897 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:25:56.931499 19897 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:25:56.936493 19897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Generated new TSK 0
I20260924 15:25:56.936663 19897 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:25:56.964831 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35221
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:57.046553 19903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:57.046782 19903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:57.046830 19903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:57.047989 19903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:57.048055 19903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:57.048103 19903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:25:57.049595 19903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:57.049906 19903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:57.050204 19903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:57.055521 19909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:57.055526 19908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:57.055820 19903 server_base.cc:1034] running on GCE node
W20260924 15:25:57.055521 19911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:57.056061 19903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:57.056299 19903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:57.057463 19903 hybrid_clock.cc:648] HybridClock initialized: now 1790263557057457 us; error 37 us; skew 500 ppm
Sep 24 15:25:57 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263557, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:57.059973 19903 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:25:57.060393 19903 webserver.cc:460] Webserver started at http://127.4.78.65:45499/ using document root <none> and password file <none>
I20260924 15:25:57.060635 19903 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:57.060714 19903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:57.060868 19903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:57.061988 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "2b480a730bbe40c58ff4ecd424b11148"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "14f856c852bb977781b9b3fb2560ec1a"
server_key_iv: "30f582c551cfb5abdb5162c6c0be5382"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.062353 19903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "2b480a730bbe40c58ff4ecd424b11148"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "14f856c852bb977781b9b3fb2560ec1a"
server_key_iv: "30f582c551cfb5abdb5162c6c0be5382"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.063647 19903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:25:57.064356 19918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.064525 19903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:25:57.064622 19903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "2b480a730bbe40c58ff4ecd424b11148"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "14f856c852bb977781b9b3fb2560ec1a"
server_key_iv: "30f582c551cfb5abdb5162c6c0be5382"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.064713 19903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:57.072513 19903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:57.074339 19903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:57.074492 19903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:57.074719 19903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:57.075016 19903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:57.075071 19903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.075112 19903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:57.075140 19903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.080878 19903 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:32911
I20260924 15:25:57.080976 20031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:32911 every 8 connection(s)
I20260924 15:25:57.081254 19903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:25:57 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263557, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:57.090044 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 19903
I20260924 15:25:57.090119 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:25:57.090322 4409 external_mini_cluster.cc:1428] Setting key 3ed27ce27891bd5dab9399d10f4ac630
I20260924 15:25:57.090449 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.082012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:54049 (local address 127.4.78.126:35221)
0924 15:25:57.082054 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.082056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.082313 (+ 257us) server_negotiation.cc:407] Connection header received
0924 15:25:57.082770 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.082771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.082780 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.082843 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:57.083869 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.084423 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.085435 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.085634 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.085689 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.087828 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:57.087830 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:57.087830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:57.087836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:57.089073 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:57.089457 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:57.089458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:57.089458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:57.089509 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:57.089721 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:57.089721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:57.089721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:57.089883 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:57.089952 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.090250 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.090353 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:25:57.090582 20034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.081986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:54049)
0924 15:25:57.082150 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.082191 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:57.082691 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:57.082895 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:57.082898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:57.083515 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:57.083772 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.083779 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.084512 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.084513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.085364 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.085367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.085736 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.085737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.085903 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.086388 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:57.086396 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:57.087735 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:57.089154 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:57.089161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:57.089163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:57.089393 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:57.089566 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:57.089567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:57.089567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:57.089662 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:57.089953 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:57.089957 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:57.090068 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 15:25:57.090190 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 15:25:57.090357 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260924 15:25:57.091064 20032 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35221
I20260924 15:25:57.091174 20032 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:57.091354 20032 heartbeater.cc:511] Master 127.4.78.126:35221 requested a full tablet report, sending...
I20260924 15:25:57.091894 19841 ts_manager.cc:194] Registered new tserver with Master: 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911)
I20260924 15:25:57.092665 19841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:54049
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:25:57.115049 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35221
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:57.195487 20037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:57.195683 20037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:57.195706 20037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:57.196791 20037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:57.196823 20037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:57.196849 20037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:25:57.198223 20037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:57.198405 20037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:57.198572 20037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:57.203526 20042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:57.203536 20045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:57.203536 20043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:57.203918 20037 server_base.cc:1034] running on GCE node
I20260924 15:25:57.204136 20037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:57.204416 20037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:57.205574 20037 hybrid_clock.cc:648] HybridClock initialized: now 1790263557205566 us; error 29 us; skew 500 ppm
Sep 24 15:25:57 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263557, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:57.208065 20037 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:25:57.208559 20037 webserver.cc:460] Webserver started at http://127.4.78.66:41399/ using document root <none> and password file <none>
I20260924 15:25:57.208837 20037 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:57.208896 20037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:57.209049 20037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:57.210721 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "e213ed048adf4b37b0f37feabb02b2cb"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "e6f4f0b2382749bea4caa4015e8a4af4"
server_key_iv: "c606a63c20ae0e13cdfa37ce948234e0"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.211200 20037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "e213ed048adf4b37b0f37feabb02b2cb"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "e6f4f0b2382749bea4caa4015e8a4af4"
server_key_iv: "c606a63c20ae0e13cdfa37ce948234e0"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.212657 20037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:25:57.213747 20052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.213961 20037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:25:57.214046 20037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "e213ed048adf4b37b0f37feabb02b2cb"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "e6f4f0b2382749bea4caa4015e8a4af4"
server_key_iv: "c606a63c20ae0e13cdfa37ce948234e0"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.214123 20037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:57.229614 20037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:57.231526 20037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:57.231698 20037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:57.231937 20037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:57.232256 20037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:57.232312 20037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.232354 20037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:57.232383 20037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.238001 20037 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:45693
I20260924 15:25:57.238077 20165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:45693 every 8 connection(s)
I20260924 15:25:57.238402 20037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:25:57.239640 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 20037
I20260924 15:25:57.239733 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:25:57.239928 4409 external_mini_cluster.cc:1428] Setting key ccdeda98120d63948ee08e2b74a060de
Sep 24 15:25:57 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263557, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
I20260924 15:25:57.249799 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.239150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:33215 (local address 127.4.78.126:35221)
0924 15:25:57.239297 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.239300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.239534 (+ 234us) server_negotiation.cc:407] Connection header received
0924 15:25:57.240253 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.240255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.240265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.240353 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:57.241646 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.242372 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.243781 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.244131 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.244250 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.246692 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:57.246694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:57.246695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:57.246703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:57.248326 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:57.248777 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:57.248779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:57.248779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:57.248841 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:57.249088 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:57.249089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:57.249090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:57.249253 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:57.249335 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.249662 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.249733 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 15:25:57.250077 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.239126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:33215)
0924 15:25:57.239347 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.239406 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:57.240143 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:57.240421 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:57.240424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:57.241259 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:57.241545 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.241551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.242449 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.242450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.243690 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.243693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.244274 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.244274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.244373 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.244908 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:57.244916 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:57.246599 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:57.248430 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:57.248436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:57.248438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:57.248690 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:57.248917 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:57.248918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:57.248919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:57.249020 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:57.249362 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:57.249370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:57.249508 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 15:25:57.249603 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 15:25:57.249727 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":103,"threads_started":1}
I20260924 15:25:57.250676 20166 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35221
I20260924 15:25:57.250801 20166 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:57.251008 20166 heartbeater.cc:511] Master 127.4.78.126:35221 requested a full tablet report, sending...
I20260924 15:25:57.251511 19841 ts_manager.cc:194] Registered new tserver with Master: e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693)
I20260924 15:25:57.251991 19841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:33215
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:25:57.268600 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35221
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:25:57.350435 20171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:25:57.350654 20171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:25:57.350679 20171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:25:57.351773 20171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:25:57.351806 20171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:25:57.351832 20171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:25:57.353250 20171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:25:57.353431 20171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:25:57.353598 20171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:25:57.358346 20179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:57.358373 20171 server_base.cc:1034] running on GCE node
W20260924 15:25:57.358434 20177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:25:57.358553 20176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:25:57.358816 20171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:25:57.359045 20171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:25:57.360203 20171 hybrid_clock.cc:648] HybridClock initialized: now 1790263557360187 us; error 46 us; skew 500 ppm
Sep 24 15:25:57 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263557, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:25:57.362537 20171 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:25:57.362914 20171 webserver.cc:460] Webserver started at http://127.4.78.67:45571/ using document root <none> and password file <none>
I20260924 15:25:57.363157 20171 fs_manager.cc:359] Metadata directory not provided
I20260924 15:25:57.363226 20171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:25:57.363377 20171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:25:57.364356 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "3e460be5cd6f458fbd1f9e5ef13af83c"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "1c68932bebda7e204eef69f1152ad610"
server_key_iv: "22ddf94d4fa47002b5ad971f43597865"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.364698 20171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "3e460be5cd6f458fbd1f9e5ef13af83c"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "1c68932bebda7e204eef69f1152ad610"
server_key_iv: "22ddf94d4fa47002b5ad971f43597865"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.366003 20171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:25:57.366696 20186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.366879 20171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:25:57.366976 20171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "3e460be5cd6f458fbd1f9e5ef13af83c"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "1c68932bebda7e204eef69f1152ad610"
server_key_iv: "22ddf94d4fa47002b5ad971f43597865"
server_key_version: "encryptionkey@0"
I20260924 15:25:57.367051 20171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:25:57.377619 20171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:25:57.379963 20171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:25:57.380117 20171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:25:57.380333 20171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:25:57.380621 20171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:25:57.380674 20171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.380714 20171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:25:57.380741 20171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:25:57.386971 20171 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:40603
I20260924 15:25:57.387014 20299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:40603 every 8 connection(s)
I20260924 15:25:57.387327 20171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:25:57 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263557, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:25:57.393487 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 20171
I20260924 15:25:57.393567 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:25:57.393785 4409 external_mini_cluster.cc:1428] Setting key 3642b901c1f0540a64c543db3f00fc3a
I20260924 15:25:57.396636 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.387986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:59549 (local address 127.4.78.126:35221)
0924 15:25:57.388025 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.388027 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.388248 (+ 221us) server_negotiation.cc:407] Connection header received
0924 15:25:57.388715 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.388716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.388725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.388786 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:57.389730 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.390300 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.391274 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.391476 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.391551 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.393737 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:57.393738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:57.393739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:57.393746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:57.395418 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:57.395832 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:57.395833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:57.395833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:57.395905 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:57.396123 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:57.396123 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:57.396124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:57.396230 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:57.396281 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.396510 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.396574 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:25:57.396819 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.387959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:59549)
0924 15:25:57.388112 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.388154 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:57.388626 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:57.388877 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:57.388880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:57.389433 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:25:57.389624 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.389628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.390382 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.390383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.391178 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.391180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.391566 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.391567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.391672 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.392205 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:25:57.392218 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:25:57.393644 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:25:57.395491 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:57.395496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:57.395498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:57.395719 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:57.395972 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:25:57.395973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:25:57.395973 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:25:57.396065 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:25:57.396298 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:25:57.396301 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:25:57.396417 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 15:25:57.396496 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:25:57.396615 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260924 15:25:57.397269 20300 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35221
I20260924 15:25:57.397392 20300 heartbeater.cc:464] Registering TS with master...
I20260924 15:25:57.397568 20300 heartbeater.cc:511] Master 127.4.78.126:35221 requested a full tablet report, sending...
I20260924 15:25:57.397948 19841 ts_manager.cc:194] Registered new tserver with Master: 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603)
I20260924 15:25:57.398291 19841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:59549
I20260924 15:25:57.405741 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:25:57.412837 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.407172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.4.78.126:35221)
0924 15:25:57.407223 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.407225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.407313 (+ 88us) server_negotiation.cc:407] Connection header received
0924 15:25:57.407366 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.407367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.407375 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.407426 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:25:57.407934 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.408396 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.409076 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.409263 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.409314 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.410544 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:25:57.410546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:25:57.410546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:25:57.410553 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:25:57.411649 (+ 1096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:57.412025 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:57.412026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:57.412026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:57.412083 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:25:57.412280 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:25:57.412281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:25:57.412282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:25:57.412426 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:25:57.412536 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.412655 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.412738 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:25:57.414458 19841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:25:57.416127 19841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:25:57.422461 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.418478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40603 (local address 127.0.0.1:42222)
0924 15:25:57.419225 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.419234 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:57.419285 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:57.419459 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:57.419460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:57.419462 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:57.419528 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.419529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.420369 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.420370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.421341 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.421344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.422160 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.422161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.422250 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.422262 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 15:25:57.422339 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:25:57.422389 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":1,"thread_start_us":43,"threads_started":1}
I20260924 15:25:57.422705 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.418585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42222 (local address 127.4.78.67:40603)
0924 15:25:57.418764 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.418769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.419286 (+ 517us) server_negotiation.cc:407] Connection header received
0924 15:25:57.419365 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.419368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.419384 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.419463 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:57.419580 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.420230 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.421425 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.422000 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.422067 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.422125 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.422579 (+ 454us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.422627 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":90,"threads_started":1}
I20260924 15:25:57.422966 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.418606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49500 (local address 127.4.78.65:32911)
0924 15:25:57.419083 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.419088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.419188 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:25:57.419429 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.419432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.419446 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.419523 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:57.419723 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.420751 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.422038 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.422531 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.422602 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.422659 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.422854 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.422897 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":93,"threads_started":1}
I20260924 15:25:57.423179 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.418857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.4.78.66:45693)
0924 15:25:57.419014 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.419019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.419031 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:25:57.419077 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.419079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.419096 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.419178 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:57.419317 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.420426 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.421891 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.422711 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.422788 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.422852 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.423051 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.423095 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260924 15:25:57.423199 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.418271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:45693 (local address 127.0.0.1:36694)
0924 15:25:57.418590 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.418635 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:57.418701 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:57.419183 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:57.419186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:57.419189 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:57.419256 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.419258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.420542 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.420543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.421792 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.421794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.422889 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.422891 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.422954 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.422958 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:25:57.423017 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:25:57.423105 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":29,"thread_start_us":58,"threads_started":1}
I20260924 15:25:57.423354 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.418273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:32911 (local address 127.0.0.1:49500)
0924 15:25:57.419129 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.419138 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:57.419354 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:57.419603 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:57.419604 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:57.419607 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:57.419645 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.419646 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.420859 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.420860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.421944 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.421946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.422691 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.422692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.422755 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.422759 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:25:57.423063 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 15:25:57.423107 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":52,"threads_started":1}
I20260924 15:25:57.423491 20234 tablet_service.cc:1467] Processing CreateTablet for tablet 36c1a6106cb24ee6afec3fb480b2d3cb (DEFAULT_TABLE table=test-table [id=7aacedc354f143d4894d997603a120ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:57.423837 20234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c1a6106cb24ee6afec3fb480b2d3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:57.423887 20100 tablet_service.cc:1467] Processing CreateTablet for tablet 36c1a6106cb24ee6afec3fb480b2d3cb (DEFAULT_TABLE table=test-table [id=7aacedc354f143d4894d997603a120ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:57.423967 19966 tablet_service.cc:1467] Processing CreateTablet for tablet 36c1a6106cb24ee6afec3fb480b2d3cb (DEFAULT_TABLE table=test-table [id=7aacedc354f143d4894d997603a120ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:25:57.424185 20100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c1a6106cb24ee6afec3fb480b2d3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:57.424211 19966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36c1a6106cb24ee6afec3fb480b2d3cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:25:57.426944 20314 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Bootstrap starting.
I20260924 15:25:57.427251 20315 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: Bootstrap starting.
I20260924 15:25:57.427311 20316 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Bootstrap starting.
I20260924 15:25:57.427680 20314 tablet_bootstrap.cc:654] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:57.427944 20316 tablet_bootstrap.cc:654] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:57.427939 20315 tablet_bootstrap.cc:654] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: Neither blocks nor log segments found. Creating new log.
I20260924 15:25:57.428337 20314 log.cc:824] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:57.428462 20316 log.cc:824] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:57.428581 20315 log.cc:824] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:25:57.429147 20316 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: No bootstrap required, opened a new log
I20260924 15:25:57.429147 20314 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: No bootstrap required, opened a new log
I20260924 15:25:57.429219 20316 ts_tablet_manager.cc:1397] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:25:57.429219 20314 ts_tablet_manager.cc:1397] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:25:57.429292 20315 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: No bootstrap required, opened a new log
I20260924 15:25:57.429369 20315 ts_tablet_manager.cc:1397] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:25:57.430732 20315 raft_consensus.cc:348] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.430892 20315 raft_consensus.cc:374] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:57.430936 20315 raft_consensus.cc:729] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e460be5cd6f458fbd1f9e5ef13af83c, State: Initialized, Role: FOLLOWER
I20260924 15:25:57.431043 20315 consensus_queue.cc:260] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.431173 20314 raft_consensus.cc:348] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.431173 20316 raft_consensus.cc:348] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.431324 20315 ts_tablet_manager.cc:1428] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:25:57.431355 20316 raft_consensus.cc:374] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:57.431356 20314 raft_consensus.cc:374] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:25:57.431394 20316 raft_consensus.cc:729] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e213ed048adf4b37b0f37feabb02b2cb, State: Initialized, Role: FOLLOWER
I20260924 15:25:57.431394 20314 raft_consensus.cc:729] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b480a730bbe40c58ff4ecd424b11148, State: Initialized, Role: FOLLOWER
I20260924 15:25:57.431488 20300 heartbeater.cc:503] Master 127.4.78.126:35221 was elected leader, sending a full tablet report...
I20260924 15:25:57.431504 20316 consensus_queue.cc:260] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.431523 20314 consensus_queue.cc:260] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.431773 20316 ts_tablet_manager.cc:1428] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:25:57.431807 20166 heartbeater.cc:503] Master 127.4.78.126:35221 was elected leader, sending a full tablet report...
I20260924 15:25:57.431958 20032 heartbeater.cc:503] Master 127.4.78.126:35221 was elected leader, sending a full tablet report...
I20260924 15:25:57.432013 20314 ts_tablet_manager.cc:1428] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260924 15:25:57.488940 20167 tablet.cc:2369] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:25:57.556165 20320 raft_consensus.cc:482] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:25:57.556326 20320 raft_consensus.cc:504] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.556742 20320 leader_election.cc:288] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693), 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911)
I20260924 15:25:57.560635 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.557096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:40359 (local address 127.4.78.65:32911)
0924 15:25:57.557141 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.557143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.557361 (+ 218us) server_negotiation.cc:407] Connection header received
0924 15:25:57.557390 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.557391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.557402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.557470 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:57.557617 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.558327 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.559663 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.560299 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.560361 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.560371 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.560526 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.560565 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:25:57.560635 20313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.557036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:60159 (local address 127.4.78.66:45693)
0924 15:25:57.557086 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.557088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.557104 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:25:57.557182 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.557183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.557197 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.557269 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:25:57.557455 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.558268 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.559663 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.560255 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.560336 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.560350 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.560526 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.560566 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:25:57.560657 20302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.556969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:45693 (local address 127.4.78.67:60159)
0924 15:25:57.557054 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.557068 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:57.557116 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:57.557293 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:57.557294 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:57.557296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:57.557372 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.557374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.558341 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.558342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.559556 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.559559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.560360 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.560361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.560441 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.560442 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:25:57.560500 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:25:57.560551 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":13}
I20260924 15:25:57.560664 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.556976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:32911 (local address 127.4.78.67:40359)
0924 15:25:57.557255 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.557265 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:25:57.557319 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:25:57.557500 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:25:57.557500 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:25:57.557502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:25:57.557547 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.557548 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.558399 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.558400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.559556 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:25:57.559559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.560374 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.560374 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:25:57.560436 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.560441 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:25:57.560500 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:25:57.560551 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":92,"thread_start_us":130,"threads_started":1}
I20260924 15:25:57.561026 20120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c1a6106cb24ee6afec3fb480b2d3cb" candidate_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e213ed048adf4b37b0f37feabb02b2cb" is_pre_election: true
I20260924 15:25:57.561079 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c1a6106cb24ee6afec3fb480b2d3cb" candidate_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b480a730bbe40c58ff4ecd424b11148" is_pre_election: true
I20260924 15:25:57.561226 19986 raft_consensus.cc:2454] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e460be5cd6f458fbd1f9e5ef13af83c in term 0.
I20260924 15:25:57.561228 20120 raft_consensus.cc:2454] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e460be5cd6f458fbd1f9e5ef13af83c in term 0.
I20260924 15:25:57.561486 20187 leader_election.cc:302] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b480a730bbe40c58ff4ecd424b11148, 3e460be5cd6f458fbd1f9e5ef13af83c; no voters:
I20260924 15:25:57.561630 20320 raft_consensus.cc:2781] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:25:57.561721 20320 raft_consensus.cc:482] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:25:57.561740 20320 raft_consensus.cc:3037] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:57.562485 20320 raft_consensus.cc:504] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.562631 20320 leader_election.cc:288] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [CANDIDATE]: Term 1 election: Requested vote from peers e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693), 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911)
I20260924 15:25:57.562788 20120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c1a6106cb24ee6afec3fb480b2d3cb" candidate_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e213ed048adf4b37b0f37feabb02b2cb"
I20260924 15:25:57.562813 19986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36c1a6106cb24ee6afec3fb480b2d3cb" candidate_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b480a730bbe40c58ff4ecd424b11148"
I20260924 15:25:57.562870 20120 raft_consensus.cc:3037] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:57.562879 19986 raft_consensus.cc:3037] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:25:57.563773 20120 raft_consensus.cc:2454] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e460be5cd6f458fbd1f9e5ef13af83c in term 1.
I20260924 15:25:57.563773 19986 raft_consensus.cc:2454] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e460be5cd6f458fbd1f9e5ef13af83c in term 1.
I20260924 15:25:57.563949 20187 leader_election.cc:302] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b480a730bbe40c58ff4ecd424b11148, 3e460be5cd6f458fbd1f9e5ef13af83c; no voters:
I20260924 15:25:57.564067 20320 raft_consensus.cc:2781] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:25:57.564224 20320 raft_consensus.cc:686] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 LEADER]: Becoming Leader. State: Replica: 3e460be5cd6f458fbd1f9e5ef13af83c, State: Running, Role: LEADER
I20260924 15:25:57.564440 20320 consensus_queue.cc:237] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:25:57.565349 19840 catalog_manager.cc:5495] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67). New cstate: current_term: 1 leader_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } health_report { overall_health: HEALTHY } } }
W20260924 15:25:57.581998 20033 tablet.cc:2369] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:25:57.605575 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.603290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42230 (local address 127.4.78.67:40603)
0924 15:25:57.603349 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:25:57.603351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:25:57.603359 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:25:57.603410 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:25:57.603411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:25:57.603430 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:25:57.603487 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:25:57.603660 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.604116 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.604847 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:25:57.605027 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:25:57.605078 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:25:57.605226 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:25:57.605335 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:25:57.605393 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:25:57.605429 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 15:25:57.605445 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:25:57.607540 20120 raft_consensus.cc:1261] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 1 FOLLOWER]: Refusing update from remote peer 3e460be5cd6f458fbd1f9e5ef13af83c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:57.607928 19986 raft_consensus.cc:1261] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Refusing update from remote peer 3e460be5cd6f458fbd1f9e5ef13af83c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:25:57.607985 20320 consensus_queue.cc:1035] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:57.608268 20320 consensus_queue.cc:1035] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:25:57.610809 20327 mvcc.cc:205] Tried to move back new op lower bound from 7332919531957481472 to 7332919531784806400. Current Snapshot: MvccSnapshot[applied={T|T < 7332919531957481472}]
I20260924 15:25:57.612978 20328 mvcc.cc:205] Tried to move back new op lower bound from 7332919531957481472 to 7332919531784806400. Current Snapshot: MvccSnapshot[applied={T|T < 7332919531957481472 or (T in {7332919531957481472})}]
I20260924 15:25:57.613193 19840 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:25:57.613296 19840 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:25:57.614030 19840 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Sending DeleteTablet for 3 replicas of tablet 36c1a6106cb24ee6afec3fb480b2d3cb
I20260924 15:25:57.614439 20234 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c1a6106cb24ee6afec3fb480b2d3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:57 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:42222
I20260924 15:25:57.614439 19966 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c1a6106cb24ee6afec3fb480b2d3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:57 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:49500
I20260924 15:25:57.614459 20100 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c1a6106cb24ee6afec3fb480b2d3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:57 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:36694
I20260924 15:25:57.614758 20339 tablet_replica.cc:330] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: stopping tablet replica
I20260924 15:25:57.614766 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 19903
I20260924 15:25:57.614876 20339 raft_consensus.cc:2229] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:25:57.614975 20339 raft_consensus.cc:2258] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:57.615121 20337 tablet_replica.cc:330] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: stopping tablet replica
I20260924 15:25:57.615222 20337 raft_consensus.cc:2229] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:25:57.615329 20337 raft_consensus.cc:2258] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:25:57.615738 20337 ts_tablet_manager.cc:1905] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:57.616531 20339 ts_tablet_manager.cc:1905] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:25:57.617375 20337 ts_tablet_manager.cc:1918] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:57.617486 20337 log.cc:1196] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/36c1a6106cb24ee6afec3fb480b2d3cb
I20260924 15:25:57.617766 20337 ts_tablet_manager.cc:1939] T 36c1a6106cb24ee6afec3fb480b2d3cb P 3e460be5cd6f458fbd1f9e5ef13af83c: Deleting consensus metadata
I20260924 15:25:57.618268 19828 catalog_manager.cc:4841] TS 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603): tablet 36c1a6106cb24ee6afec3fb480b2d3cb (table test-table [id=7aacedc354f143d4894d997603a120ce]) successfully deleted
I20260924 15:25:57.618494 20339 ts_tablet_manager.cc:1918] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:25:57.618562 20339 log.cc:1196] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/36c1a6106cb24ee6afec3fb480b2d3cb
I20260924 15:25:57.618820 20339 ts_tablet_manager.cc:1939] T 36c1a6106cb24ee6afec3fb480b2d3cb P e213ed048adf4b37b0f37feabb02b2cb: Deleting consensus metadata
I20260924 15:25:57.619261 19828 catalog_manager.cc:4841] TS e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693): tablet 36c1a6106cb24ee6afec3fb480b2d3cb (table test-table [id=7aacedc354f143d4894d997603a120ce]) successfully deleted
W20260924 15:25:57.622612 19825 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:32911 (error 108)
I20260924 15:25:57.622839 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 20037
I20260924 15:25:57.623204 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:25:57.623096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:32911 (local address 127.0.0.1:49506)
0924 15:25:57.623138 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 15:25:57.623167 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 15:25:57.623304 19825 catalog_manager.cc:4584] TS 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36c1a6106cb24ee6afec3fb480b2d3cb: Network error: Client connection negotiation failed: client connection to 127.4.78.65:32911: connect: Connection refused (error 111)
I20260924 15:25:57.628237 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 20171
I20260924 15:25:57.633504 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 19809
I20260924 15:26:17.639271 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35221
--webserver_interface=127.4.78.126
--webserver_port=37487
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:35221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:17.727849 20341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:17.728066 20341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:17.728091 20341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:17.729365 20341 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:26:17.729413 20341 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:26:17.729427 20341 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:26:17.729439 20341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:26:17.729496 20341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:17.729516 20341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:17.729539 20341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:26:17.729570 20341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:26:17.731315 20341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:35221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=37487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:17.731594 20341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:17.731841 20341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:17.740923 20348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:17.740974 20347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:17.741098 20341 server_base.cc:1034] running on GCE node
W20260924 15:26:17.740916 20350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:17.741568 20341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:17.741969 20341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:17.743168 20341 hybrid_clock.cc:648] HybridClock initialized: now 1790263577743144 us; error 30 us; skew 500 ppm
Sep 24 15:26:17 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263577, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:17.755637 20341 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:26:17.756160 20341 webserver.cc:460] Webserver started at http://127.4.78.126:37487/ using document root <none> and password file <none>
I20260924 15:26:17.756407 20341 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:17.756488 20341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:17.757936 20341 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:17.758617 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:17.758807 20341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:17.758908 20341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "ef602e27976248f3ba323b4996c9487f"
format_stamp: "Formatted at 2026-09-24 15:25:56 on dist-test-slave-l766"
server_key: "e7145ab9420572df70e138c14834f948"
server_key_iv: "733fcf2fa55a941f295948361b8ac19e"
server_key_version: "encryptionkey@0"
I20260924 15:26:17.759310 20341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:17.767129 20341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:17.769377 20341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:17.769537 20341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:17.773578 20341 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:35221
I20260924 15:26:17.773609 20409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:35221 every 8 connection(s)
I20260924 15:26:17.774086 20341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:26:17.774816 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 20341
I20260924 15:26:17.775081 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:32911
--local_ip_for_outbound_sockets=127.4.78.65
--tserver_master_addrs=127.4.78.126:35221
--webserver_port=45499
--webserver_interface=127.4.78.65
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:26:17.775733 20410 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:26:17.776396 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Bootstrap starting.
I20260924 15:26:17.784533 20410 log.cc:824] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:17.786507 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:26:17.786792 20410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Bootstrap complete.
I20260924 15:26:17.789124 20410 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:26:17.789397 20410 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef602e27976248f3ba323b4996c9487f, State: Initialized, Role: FOLLOWER
I20260924 15:26:17.789592 20410 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:26:17.789673 20410 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:26:17.789738 20410 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:26:17.789810 20410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:26:17.790817 20410 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:26:17.790939 20410 leader_election.cc:302] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef602e27976248f3ba323b4996c9487f; no voters:
I20260924 15:26:17.791136 20410 leader_election.cc:288] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:26:17.791172 20414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:26:17.791299 20414 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 2 LEADER]: Becoming Leader. State: Replica: ef602e27976248f3ba323b4996c9487f, State: Running, Role: LEADER
I20260924 15:26:17.791404 20410 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:26:17.791386 20414 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:26:17.791731 20415 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ef602e27976248f3ba323b4996c9487f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } } }
I20260924 15:26:17.791746 20416 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef602e27976248f3ba323b4996c9487f. Latest consensus state: current_term: 2 leader_uuid: "ef602e27976248f3ba323b4996c9487f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } } }
I20260924 15:26:17.791833 20415 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:17.791843 20416 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:17.792035 20418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:26:17.792905 20418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:26:17.793022 20418 catalog_manager.cc:1259] Loaded cluster ID: c14ada371e5643c098d826ef5c2939cc
I20260924 15:26:17.793279 20418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:26:17.794514 20418 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:26:17.794691 20418 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Loaded TSK: 0
I20260924 15:26:17.806979 20418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Generated new TSK 1
I20260924 15:26:17.807078 20418 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:26:17.858417 20412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:17.858639 20412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:17.858680 20412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:17.859877 20412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:17.859935 20412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:17.859979 20412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:26:17.861408 20412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:32911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=45499
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:17.861675 20412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:17.861938 20412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:17.867760 20440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:17.867725 20438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:17.867717 20437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:17.868209 20412 server_base.cc:1034] running on GCE node
I20260924 15:26:17.868441 20412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:17.868698 20412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:17.869874 20412 hybrid_clock.cc:648] HybridClock initialized: now 1790263577869856 us; error 34 us; skew 500 ppm
Sep 24 15:26:17 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263577, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:17.872422 20412 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:26:17.872916 20412 webserver.cc:460] Webserver started at http://127.4.78.65:45499/ using document root <none> and password file <none>
I20260924 15:26:17.873167 20412 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:17.873246 20412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:17.874703 20412 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:17.875428 20447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:17.875613 20412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:17.875702 20412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "2b480a730bbe40c58ff4ecd424b11148"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "14f856c852bb977781b9b3fb2560ec1a"
server_key_iv: "30f582c551cfb5abdb5162c6c0be5382"
server_key_version: "encryptionkey@0"
I20260924 15:26:17.876051 20412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:17.885399 20412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:17.887719 20412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:17.887882 20412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:17.888121 20412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:17.888581 20454 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:26:17.890617 20412 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:26:17.890683 20412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:26:17.890725 20412 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:26:17.891494 20412 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:26:17.891549 20412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:26:17.891598 20454 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Bootstrap starting.
I20260924 15:26:17.898972 20412 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:32911
I20260924 15:26:17.899358 20412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:26:17.900478 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 20412
I20260924 15:26:17.900761 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:45693
--local_ip_for_outbound_sockets=127.4.78.66
--tserver_master_addrs=127.4.78.126:35221
--webserver_port=41399
--webserver_interface=127.4.78.66
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:26:17.902463 20454 log.cc:824] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:17.905154 20454 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:26:17.905526 20454 tablet_bootstrap.cc:492] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Bootstrap complete.
I20260924 15:26:17.906838 20454 ts_tablet_manager.cc:1397] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260924 15:26:17.908339 20454 raft_consensus.cc:348] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:26:17.909487 20454 raft_consensus.cc:729] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b480a730bbe40c58ff4ecd424b11148, State: Initialized, Role: FOLLOWER
I20260924 15:26:17.909711 20454 consensus_queue.cc:260] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } } peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } }
I20260924 15:26:17.910113 20454 ts_tablet_manager.cc:1428] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
Sep 24 15:26:17 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263577, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:17.913910 20561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:32911 every 8 connection(s)
I20260924 15:26:17.914199 20564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:17.900087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:58593)
0924 15:26:17.900279 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:17.900323 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:17.900995 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:17.903343 (+ 2348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:17.903347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:17.903713 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:17.903974 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:17.903981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:17.905724 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:17.905726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:17.907229 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:17.907232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:17.908114 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:17.908115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:17.908271 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:17.908898 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:17.908907 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:17.910609 (+ 1702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:17.912554 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:17.912561 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:17.912563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:17.912791 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:17.913050 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:17.913051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:17.913051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:17.913174 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:17.913558 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:17.913565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:17.913700 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 15:26:17.913805 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:26:17.913999 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260924 15:26:17.914554 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:17.900174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:58593 (local address 127.4.78.126:35221)
0924 15:26:17.902871 (+ 2697us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:17.902880 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:17.902926 (+ 46us) server_negotiation.cc:407] Connection header received
0924 15:26:17.903069 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:17.903076 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:17.903132 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:17.903288 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:17.904127 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:17.905524 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:17.907357 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:17.907990 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:17.908571 (+ 581us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:17.910730 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:17.910734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:17.910735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:17.910754 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:17.912448 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:17.912878 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:17.912882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:17.912882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:17.912978 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:17.913250 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:17.913251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:17.913251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:17.913479 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:17.913566 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:17.913894 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 15:26:17.914087 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2643,"thread_start_us":87,"threads_started":1}
I20260924 15:26:17.915203 20562 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35221
I20260924 15:26:17.915306 20562 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:17.915480 20562 heartbeater.cc:511] Master 127.4.78.126:35221 requested a full tablet report, sending...
I20260924 15:26:17.916136 20374 ts_manager.cc:194] Registered new tserver with Master: 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911)
I20260924 15:26:17.917238 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:58593
I20260924 15:26:17.918210 20562 heartbeater.cc:503] Master 127.4.78.126:35221 was elected leader, sending a full tablet report...
I20260924 15:26:17.925321 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:17.916698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:32911 (local address 127.0.0.1:48864)
0924 15:26:17.922075 (+ 5377us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:17.922119 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:17.922203 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:17.922483 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:17.922486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:17.922488 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:17.922567 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:17.922568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:17.923443 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:17.923444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:17.924264 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:17.924267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:17.925008 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:17.925009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:17.925110 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:17.925123 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:26:17.925197 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 15:26:17.925250 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5344,"thread_start_us":90,"threads_started":1}
I20260924 15:26:17.925622 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:17.916939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48864 (local address 127.4.78.65:32911)
0924 15:26:17.917103 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:17.917107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:17.922289 (+ 5182us) server_negotiation.cc:407] Connection header received
0924 15:26:17.922327 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:17.922329 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:17.922343 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:17.922413 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:17.922672 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:17.923353 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:17.924371 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:17.924903 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:17.924991 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:17.925049 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:17.925514 (+ 465us) server_negotiation.cc:299] Negotiation successful
0924 15:26:17.925557 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":89,"threads_started":1}
I20260924 15:26:17.925999 20496 tablet_service.cc:1514] Processing DeleteTablet for tablet 36c1a6106cb24ee6afec3fb480b2d3cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:25:57 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:48864
I20260924 15:26:17.926477 20572 tablet_replica.cc:330] stopping tablet replica
I20260924 15:26:17.926606 20572 raft_consensus.cc:2229] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:17.926676 20572 raft_consensus.cc:2258] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:17.926973 20572 ts_tablet_manager.cc:1905] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:17.929196 20572 ts_tablet_manager.cc:1918] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:17.929291 20572 log.cc:1196] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal/wals/36c1a6106cb24ee6afec3fb480b2d3cb
I20260924 15:26:17.929548 20572 ts_tablet_manager.cc:1939] T 36c1a6106cb24ee6afec3fb480b2d3cb P 2b480a730bbe40c58ff4ecd424b11148: Deleting consensus metadata
I20260924 15:26:17.930087 20358 catalog_manager.cc:4841] TS 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911): tablet 36c1a6106cb24ee6afec3fb480b2d3cb (table test-table [id=7aacedc354f143d4894d997603a120ce]) successfully deleted
W20260924 15:26:17.994319 20567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:17.994524 20567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:17.994547 20567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:17.995632 20567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:17.995667 20567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:17.995698 20567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:26:17.997044 20567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:45693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=41399
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:17.997216 20567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:17.997381 20567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:18.002241 20578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:18.002249 20580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:18.002241 20577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:18.002542 20567 server_base.cc:1034] running on GCE node
I20260924 15:26:18.002794 20567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:18.003028 20567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:18.004209 20567 hybrid_clock.cc:648] HybridClock initialized: now 1790263578004195 us; error 29 us; skew 500 ppm
Sep 24 15:26:18 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263578, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:18.006911 20567 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:26:18.007385 20567 webserver.cc:460] Webserver started at http://127.4.78.66:41399/ using document root <none> and password file <none>
I20260924 15:26:18.007603 20567 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:18.007663 20567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:18.008960 20567 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:26:18.009553 20587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:18.009718 20567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:26:18.009801 20567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "e213ed048adf4b37b0f37feabb02b2cb"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "e6f4f0b2382749bea4caa4015e8a4af4"
server_key_iv: "c606a63c20ae0e13cdfa37ce948234e0"
server_key_version: "encryptionkey@0"
I20260924 15:26:18.010171 20567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:18.023398 20567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:18.025242 20567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:18.025354 20567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:18.025528 20567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:18.025898 20567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:18.025936 20567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:18.025976 20567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:18.026002 20567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:18.031721 20567 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:45693
I20260924 15:26:18.031790 20700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:45693 every 8 connection(s)
I20260924 15:26:18.032152 20567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:26:18.036659 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 20567
I20260924 15:26:18.036984 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:40603
--local_ip_for_outbound_sockets=127.4.78.67
--tserver_master_addrs=127.4.78.126:35221
--webserver_port=45571
--webserver_interface=127.4.78.67
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:26:18 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263578, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:18.042255 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.032903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:54781 (local address 127.4.78.126:35221)
0924 15:26:18.032976 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.032979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.033208 (+ 229us) server_negotiation.cc:407] Connection header received
0924 15:26:18.033692 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.033693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.033708 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.033779 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:18.034807 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.035333 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.036386 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.036620 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.036712 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.039283 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:18.039285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:18.039286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:18.039296 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:18.040754 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:18.041181 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:18.041182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:18.041183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:18.041245 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:18.041515 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:18.041517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:18.041517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:18.041637 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:18.041710 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.042066 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.042163 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:26:18.042469 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.032897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:54781)
0924 15:26:18.033085 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:18.033121 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:18.033580 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:18.033909 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:18.033913 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:18.034486 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:18.034695 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.034704 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.035426 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.035427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.036299 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.036301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.036724 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.036725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.036879 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.037555 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:18.037564 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:18.039168 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:18.040858 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:18.040865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:18.040867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:18.041101 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:18.041317 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:18.041318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:18.041318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:18.041443 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:18.041714 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:18.041721 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:18.041929 (+ 208us) client_negotiation.cc:769] Sending connection context
0924 15:26:18.042043 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 15:26:18.042202 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":102,"threads_started":1}
I20260924 15:26:18.043747 20701 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35221
I20260924 15:26:18.043836 20701 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:18.044034 20701 heartbeater.cc:511] Master 127.4.78.126:35221 requested a full tablet report, sending...
I20260924 15:26:18.044458 20374 ts_manager.cc:194] Registered new tserver with Master: e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693)
I20260924 15:26:18.045023 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:54781
W20260924 15:26:18.120781 20704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:18.120980 20704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:18.121003 20704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:18.122164 20704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:18.122207 20704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:18.122233 20704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:26:18.123617 20704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:40603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=45571
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:18.123795 20704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:18.123961 20704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:18.129040 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:18.129076 20710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:18.129040 20709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:18.129307 20704 server_base.cc:1034] running on GCE node
I20260924 15:26:18.129509 20704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:18.129699 20704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:18.130831 20704 hybrid_clock.cc:648] HybridClock initialized: now 1790263578130805 us; error 37 us; skew 500 ppm
Sep 24 15:26:18 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263578, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:18.133302 20704 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:26:18.133734 20704 webserver.cc:460] Webserver started at http://127.4.78.67:45571/ using document root <none> and password file <none>
I20260924 15:26:18.133981 20704 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:18.134065 20704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:18.135414 20704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:18.136034 20719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:18.136206 20704 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:18.136310 20704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "3e460be5cd6f458fbd1f9e5ef13af83c"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "1c68932bebda7e204eef69f1152ad610"
server_key_iv: "22ddf94d4fa47002b5ad971f43597865"
server_key_version: "encryptionkey@0"
I20260924 15:26:18.136713 20704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:18.154054 20704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:18.155939 20704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:18.156117 20704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:18.156364 20704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:18.156689 20704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:18.156752 20704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:18.156793 20704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:18.156833 20704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:18.162906 20704 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:40603
I20260924 15:26:18.162978 20832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:40603 every 8 connection(s)
I20260924 15:26:18.163309 20704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:26:18 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263578, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:18.172422 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.164066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:48033 (local address 127.4.78.126:35221)
0924 15:26:18.164124 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.164126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.164394 (+ 268us) server_negotiation.cc:407] Connection header received
0924 15:26:18.164938 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.164940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.164952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.165007 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:18.165972 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.166478 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.167464 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.167676 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.167762 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.169768 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:18.169770 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:18.169770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:18.169778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:18.171048 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:18.171451 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:18.171452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:18.171452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:18.171515 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:18.171754 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:18.171755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:18.171756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:18.171877 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:18.171950 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.172225 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.172314 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:26:18.172588 20835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.164057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:48033)
0924 15:26:18.164263 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:18.164312 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:18.164817 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:18.165083 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:18.165087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:18.165632 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:18.165877 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.165883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.166573 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.166574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.167373 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.167374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.167770 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.167772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.167897 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.168349 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:18.168357 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:18.169673 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:18.171145 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:18.171152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:18.171154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:18.171376 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:18.171584 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:18.171585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:18.171586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:18.171686 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:18.171956 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:18.171962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:18.172095 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 15:26:18.172203 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 15:26:18.172355 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":121,"threads_started":1}
I20260924 15:26:18.172708 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 20704
I20260924 15:26:18.173148 20833 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35221
I20260924 15:26:18.173246 20833 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:18.173456 20833 heartbeater.cc:511] Master 127.4.78.126:35221 requested a full tablet report, sending...
I20260924 15:26:18.173929 20374 ts_manager.cc:194] Registered new tserver with Master: 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603)
I20260924 15:26:18.174367 20374 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:48033
I20260924 15:26:18.179749 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.173212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.4.78.126:35221)
0924 15:26:18.173255 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.173257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.173481 (+ 224us) server_negotiation.cc:407] Connection header received
0924 15:26:18.173575 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.173576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.173585 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.173652 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:18.174354 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.174870 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.175659 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.175841 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.175894 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.177038 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:18.177039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:18.177040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:18.177047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:18.178711 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:18.179084 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:18.179085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:18.179086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:18.179142 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:18.179299 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:18.179300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:18.179300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:18.179436 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:18.179551 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.179578 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.179643 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:26:18.180539 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:26:18.183104 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.180886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39378 (local address 127.4.78.126:35221)
0924 15:26:18.180941 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.180943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.181039 (+ 96us) server_negotiation.cc:407] Connection header received
0924 15:26:18.181109 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.181110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.181137 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.181193 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:26:18.181277 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.181708 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.182492 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.182661 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.182711 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.182882 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:26:18.182925 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 15:26:18.183063 (+ 138us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39378: token expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 15:26:18.183182 20565 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39378: token expired
W20260924 15:26:18.183349 20304 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.78.126:35221)
I20260924 15:26:18.188694 20565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.183580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39380 (local address 127.4.78.126:35221)
0924 15:26:18.183638 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.183640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.183661 (+ 21us) server_negotiation.cc:407] Connection header received
0924 15:26:18.183711 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.183712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.183719 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.183764 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:18.184169 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.184628 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.185286 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.185461 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.185517 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.186631 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:18.186633 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:18.186633 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:18.186639 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:18.187766 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:18.188055 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:18.188056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:18.188056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:18.188103 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:18.188270 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:18.188271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:18.188271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:18.188408 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:18.188494 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.188580 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.188638 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:26:18.189848 20374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39380:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:26:18.191541 20374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:26:18.193611 20496 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8b3e681e9040f19b2c312f6cfae246 (DEFAULT_TABLE table=test-table [id=2a6487d3c924465b88491feff142449d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:18.193866 20496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8b3e681e9040f19b2c312f6cfae246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:18.195021 20454 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Bootstrap starting.
I20260924 15:26:18.195341 20454 tablet_bootstrap.cc:654] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:18.195916 20454 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: No bootstrap required, opened a new log
I20260924 15:26:18.195968 20454 ts_tablet_manager.cc:1397] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:26:18.196065 20454 raft_consensus.cc:348] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.196134 20454 raft_consensus.cc:374] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:18.196201 20454 raft_consensus.cc:729] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b480a730bbe40c58ff4ecd424b11148, State: Initialized, Role: FOLLOWER
I20260924 15:26:18.196251 20454 consensus_queue.cc:260] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.196331 20454 ts_tablet_manager.cc:1428] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:18.197626 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.193403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40603 (local address 127.0.0.1:59254)
0924 15:26:18.193441 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:18.193452 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:18.193486 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:18.194074 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:18.194075 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:18.194077 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:18.194118 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.194119 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.195321 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.195322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.196389 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.196393 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.197398 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.197399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.197458 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.197462 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:18.197520 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:26:18.197563 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 15:26:18.197645 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.193608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.4.78.67:40603)
0924 15:26:18.193889 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.193894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.193906 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:26:18.193947 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.193949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.193965 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.194065 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:18.194179 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.195205 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.196655 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.197310 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.197399 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.197459 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.197534 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.197570 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
I20260924 15:26:18.198289 20767 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8b3e681e9040f19b2c312f6cfae246 (DEFAULT_TABLE table=test-table [id=2a6487d3c924465b88491feff142449d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:18.198544 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.193492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:45693 (local address 127.0.0.1:35940)
0924 15:26:18.195052 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:18.195064 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:18.195117 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:18.195325 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:18.195325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:18.195328 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:18.195373 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.195375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.196585 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.196586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.197445 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.197448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.198311 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.198312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.198377 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.198381 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:18.198439 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:26:18.198477 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1546,"thread_start_us":59,"threads_started":1}
I20260924 15:26:18.198567 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.193590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.4.78.66:45693)
0924 15:26:18.193891 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.193894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.195175 (+ 1281us) server_negotiation.cc:407] Connection header received
0924 15:26:18.195222 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.195229 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.195242 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.195328 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:18.195438 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.196506 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.197535 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.198206 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.198305 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.198359 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.198452 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.198491 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":123,"threads_started":1}
I20260924 15:26:18.198669 20767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8b3e681e9040f19b2c312f6cfae246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:18.199271 20635 tablet_service.cc:1467] Processing CreateTablet for tablet 7c8b3e681e9040f19b2c312f6cfae246 (DEFAULT_TABLE table=test-table [id=2a6487d3c924465b88491feff142449d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:18.199518 20635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c8b3e681e9040f19b2c312f6cfae246. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:18.201581 20842 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: Bootstrap starting.
I20260924 15:26:18.202014 20843 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Bootstrap starting.
I20260924 15:26:18.202433 20842 tablet_bootstrap.cc:654] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:18.202706 20843 tablet_bootstrap.cc:654] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:18.203080 20842 log.cc:824] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:18.203312 20843 log.cc:824] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:18.204018 20842 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: No bootstrap required, opened a new log
I20260924 15:26:18.204018 20843 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: No bootstrap required, opened a new log
I20260924 15:26:18.204092 20843 ts_tablet_manager.cc:1397] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:26:18.204092 20842 ts_tablet_manager.cc:1397] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260924 15:26:18.206049 20843 raft_consensus.cc:348] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.206035 20842 raft_consensus.cc:348] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.206228 20843 raft_consensus.cc:374] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:18.206228 20842 raft_consensus.cc:374] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:18.206266 20843 raft_consensus.cc:729] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e213ed048adf4b37b0f37feabb02b2cb, State: Initialized, Role: FOLLOWER
I20260924 15:26:18.206274 20842 raft_consensus.cc:729] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e460be5cd6f458fbd1f9e5ef13af83c, State: Initialized, Role: FOLLOWER
I20260924 15:26:18.206395 20843 consensus_queue.cc:260] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.206395 20842 consensus_queue.cc:260] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.206722 20842 ts_tablet_manager.cc:1428] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:26:18.206722 20843 ts_tablet_manager.cc:1428] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:26:18.206761 20701 heartbeater.cc:503] Master 127.4.78.126:35221 was elected leader, sending a full tablet report...
I20260924 15:26:18.206856 20833 heartbeater.cc:503] Master 127.4.78.126:35221 was elected leader, sending a full tablet report...
W20260924 15:26:18.282709 20702 tablet.cc:2369] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:26:18.350263 20846 raft_consensus.cc:482] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:18.350399 20846 raft_consensus.cc:504] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.350754 20846 leader_election.cc:288] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603), 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911)
I20260924 15:26:18.354005 20703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.350951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40603 (local address 127.4.78.66:36547)
0924 15:26:18.351010 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:18.351024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:18.351069 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:18.351285 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:18.351286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:18.351288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:18.351373 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.351375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.352163 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.352164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.352984 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.352986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.353678 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.353680 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.353784 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.353788 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:18.353866 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:26:18.353932 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 15:26:18.354068 20839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.351077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:36547 (local address 127.4.78.67:40603)
0924 15:26:18.351127 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.351129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.351141 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:26:18.351174 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.351176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.351202 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.351262 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:18.351461 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.351978 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.353090 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.353562 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.353623 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.353632 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.353924 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.353973 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:26:18.354205 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.350973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:32911 (local address 127.4.78.66:39549)
0924 15:26:18.351160 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:18.351169 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:18.351234 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:18.351395 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:18.351396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:18.351397 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:18.351487 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.351489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.352316 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.352316 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.353111 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:18.353113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.353916 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.353917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:18.353997 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.354002 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:26:18.354065 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:26:18.354111 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260924 15:26:18.354244 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.351055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:39549 (local address 127.4.78.65:32911)
0924 15:26:18.351120 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.351122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.351280 (+ 158us) server_negotiation.cc:407] Connection header received
0924 15:26:18.351308 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.351309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.351321 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.351375 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:18.351583 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.352207 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.353229 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.353662 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.353754 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.353766 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.354088 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.354158 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:26:18.354432 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8b3e681e9040f19b2c312f6cfae246" candidate_uuid: "e213ed048adf4b37b0f37feabb02b2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" is_pre_election: true
I20260924 15:26:18.354600 20787 raft_consensus.cc:2454] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e213ed048adf4b37b0f37feabb02b2cb in term 0.
I20260924 15:26:18.354784 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8b3e681e9040f19b2c312f6cfae246" candidate_uuid: "e213ed048adf4b37b0f37feabb02b2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b480a730bbe40c58ff4ecd424b11148" is_pre_election: true
I20260924 15:26:18.354863 20591 leader_election.cc:302] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e460be5cd6f458fbd1f9e5ef13af83c, e213ed048adf4b37b0f37feabb02b2cb; no voters:
I20260924 15:26:18.354964 20516 raft_consensus.cc:2454] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e213ed048adf4b37b0f37feabb02b2cb in term 0.
I20260924 15:26:18.355054 20846 raft_consensus.cc:2781] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:26:18.355154 20846 raft_consensus.cc:482] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:26:18.355182 20846 raft_consensus.cc:3037] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:18.355921 20846 raft_consensus.cc:504] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.356200 20846 leader_election.cc:288] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [CANDIDATE]: Term 1 election: Requested vote from peers 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603), 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911)
I20260924 15:26:18.356356 20787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8b3e681e9040f19b2c312f6cfae246" candidate_uuid: "e213ed048adf4b37b0f37feabb02b2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c"
I20260924 15:26:18.356348 20516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c8b3e681e9040f19b2c312f6cfae246" candidate_uuid: "e213ed048adf4b37b0f37feabb02b2cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b480a730bbe40c58ff4ecd424b11148"
I20260924 15:26:18.356419 20787 raft_consensus.cc:3037] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:18.356424 20516 raft_consensus.cc:3037] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:18.357084 20516 raft_consensus.cc:2454] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e213ed048adf4b37b0f37feabb02b2cb in term 1.
I20260924 15:26:18.357115 20787 raft_consensus.cc:2454] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e213ed048adf4b37b0f37feabb02b2cb in term 1.
I20260924 15:26:18.357272 20591 leader_election.cc:302] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e460be5cd6f458fbd1f9e5ef13af83c, e213ed048adf4b37b0f37feabb02b2cb; no voters:
I20260924 15:26:18.357384 20846 raft_consensus.cc:2781] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:18.357553 20846 raft_consensus.cc:686] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 1 LEADER]: Becoming Leader. State: Replica: e213ed048adf4b37b0f37feabb02b2cb, State: Running, Role: LEADER
I20260924 15:26:18.357656 20846 consensus_queue.cc:237] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:18.358614 20373 catalog_manager.cc:5495] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb reported cstate change: term changed from 0 to 1, leader changed from <none> to e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66). New cstate: current_term: 1 leader_uuid: "e213ed048adf4b37b0f37feabb02b2cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } health_report { overall_health: UNKNOWN } } }
I20260924 15:26:18.383168 20840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.380613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.4.78.66:45693)
0924 15:26:18.380698 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:18.380700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:18.380709 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:26:18.380749 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:18.380750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:18.380769 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:18.380848 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:26:18.381022 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.381486 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.382363 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:18.382563 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:18.382631 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:18.382817 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:26:18.382932 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:26:18.382998 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:18.383060 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:26:18.383080 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:26:18.385097 20787 raft_consensus.cc:1261] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 FOLLOWER]: Refusing update from remote peer e213ed048adf4b37b0f37feabb02b2cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:18.385097 20516 raft_consensus.cc:1261] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Refusing update from remote peer e213ed048adf4b37b0f37feabb02b2cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:18.385468 20849 consensus_queue.cc:1035] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:18.385634 20846 consensus_queue.cc:1035] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:18.388168 20853 mvcc.cc:205] Tried to move back new op lower bound from 7332919617062465536 to 7332919616953819136. Current Snapshot: MvccSnapshot[applied={T|T < 7332919617062465536}]
I20260924 15:26:18.390352 20374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:18.390460 20374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39380:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:18.391247 20374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Sending DeleteTablet for 3 replicas of tablet 7c8b3e681e9040f19b2c312f6cfae246
I20260924 15:26:18.391574 20767 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8b3e681e9040f19b2c312f6cfae246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:18 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:59254
I20260924 15:26:18.391601 20496 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8b3e681e9040f19b2c312f6cfae246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:18 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:48864
I20260924 15:26:18.391647 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 20412
I20260924 15:26:18.391714 20572 tablet_replica.cc:330] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: stopping tablet replica
I20260924 15:26:18.391724 20635 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c8b3e681e9040f19b2c312f6cfae246 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:18 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:35940
I20260924 15:26:18.391780 20572 raft_consensus.cc:2229] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:18.391830 20572 raft_consensus.cc:2258] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:18.391999 20863 tablet_replica.cc:330] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: stopping tablet replica
I20260924 15:26:18.392086 20863 raft_consensus.cc:2229] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:26:18.392230 20862 tablet_replica.cc:330] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: stopping tablet replica
I20260924 15:26:18.392362 20862 raft_consensus.cc:2229] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:18.392464 20862 raft_consensus.cc:2258] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:18.392170 20863 raft_consensus.cc:2258] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:18.392951 20862 ts_tablet_manager.cc:1905] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:18.393121 20863 ts_tablet_manager.cc:1905] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:18.395130 20863 ts_tablet_manager.cc:1918] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:18.395130 20862 ts_tablet_manager.cc:1918] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:18.395207 20863 log.cc:1196] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/7c8b3e681e9040f19b2c312f6cfae246
I20260924 15:26:18.395321 20862 log.cc:1196] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/7c8b3e681e9040f19b2c312f6cfae246
I20260924 15:26:18.395505 20863 ts_tablet_manager.cc:1939] T 7c8b3e681e9040f19b2c312f6cfae246 P e213ed048adf4b37b0f37feabb02b2cb: Deleting consensus metadata
I20260924 15:26:18.395744 20862 ts_tablet_manager.cc:1939] T 7c8b3e681e9040f19b2c312f6cfae246 P 3e460be5cd6f458fbd1f9e5ef13af83c: Deleting consensus metadata
I20260924 15:26:18.395954 20361 catalog_manager.cc:4841] TS e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693): tablet 7c8b3e681e9040f19b2c312f6cfae246 (table test-table [id=2a6487d3c924465b88491feff142449d]) successfully deleted
I20260924 15:26:18.396227 20361 catalog_manager.cc:4841] TS 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603): tablet 7c8b3e681e9040f19b2c312f6cfae246 (table test-table [id=2a6487d3c924465b88491feff142449d]) successfully deleted
W20260924 15:26:18.398003 20358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:32911 (error 108)
I20260924 15:26:18.398509 20838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:18.398376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:32911 (local address 127.0.0.1:48874)
0924 15:26:18.398436 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:18.398475 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:32911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:26:18.398576 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 20567
W20260924 15:26:18.400540 20358 catalog_manager.cc:4584] TS 2b480a730bbe40c58ff4ecd424b11148 (127.4.78.65:32911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7c8b3e681e9040f19b2c312f6cfae246: Network error: Client connection negotiation failed: client connection to 127.4.78.65:32911: connect: Connection refused (error 111)
I20260924 15:26:18.404754 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 20704
I20260924 15:26:18.409637 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 20341
I20260924 15:26:38.415288 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35221
--webserver_interface=127.4.78.126
--webserver_port=37487
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:35221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:38.496256 20865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:38.496503 20865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:38.496541 20865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:38.497710 20865 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:26:38.497783 20865 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:26:38.497805 20865 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:26:38.497823 20865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:26:38.497871 20865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:38.497901 20865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:38.497931 20865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:26:38.497958 20865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:26:38.499434 20865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:35221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=37487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:38.499713 20865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:38.499977 20865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:38.504815 20870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:38.504830 20873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:38.504830 20871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:38.504894 20865 server_base.cc:1034] running on GCE node
I20260924 15:26:38.505234 20865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:38.505571 20865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:38.506757 20865 hybrid_clock.cc:648] HybridClock initialized: now 1790263598506717 us; error 30 us; skew 500 ppm
Sep 24 15:26:38 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263598, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:38.514614 20865 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:26:38.515156 20865 webserver.cc:460] Webserver started at http://127.4.78.126:37487/ using document root <none> and password file <none>
I20260924 15:26:38.515398 20865 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:38.515471 20865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:38.516938 20865 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:38.517635 20880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:38.517899 20865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:38.518016 20865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "ef602e27976248f3ba323b4996c9487f"
format_stamp: "Formatted at 2026-09-24 15:25:56 on dist-test-slave-l766"
server_key: "e7145ab9420572df70e138c14834f948"
server_key_iv: "733fcf2fa55a941f295948361b8ac19e"
server_key_version: "encryptionkey@0"
I20260924 15:26:38.518395 20865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:38.541462 20865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:38.544091 20865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:38.544277 20865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:38.548640 20865 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:35221
I20260924 15:26:38.548714 20932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:35221 every 8 connection(s)
I20260924 15:26:38.549118 20865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:26:38.550589 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 20865
I20260924 15:26:38.550920 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:32911
--local_ip_for_outbound_sockets=127.4.78.65
--tserver_master_addrs=127.4.78.126:35221
--webserver_port=45499
--webserver_interface=127.4.78.65
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:26:38.551030 20933 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:26:38.552130 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Bootstrap starting.
I20260924 15:26:38.560634 20933 log.cc:824] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:38.562958 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:26:38.563231 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Bootstrap complete.
I20260924 15:26:38.565505 20933 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:26:38.565726 20933 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef602e27976248f3ba323b4996c9487f, State: Initialized, Role: FOLLOWER
I20260924 15:26:38.565897 20933 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:26:38.565977 20933 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:26:38.566006 20933 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:26:38.566071 20933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 2 FOLLOWER]: Advancing to term 3
I20260924 15:26:38.567062 20933 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:26:38.567183 20933 leader_election.cc:302] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef602e27976248f3ba323b4996c9487f; no voters:
I20260924 15:26:38.567390 20933 leader_election.cc:288] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 15:26:38.567425 20937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 3 FOLLOWER]: Leader election won for term 3
I20260924 15:26:38.567543 20937 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [term 3 LEADER]: Becoming Leader. State: Replica: ef602e27976248f3ba323b4996c9487f, State: Running, Role: LEADER
I20260924 15:26:38.567629 20937 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } }
I20260924 15:26:38.567659 20933 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:26:38.568279 20938 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ef602e27976248f3ba323b4996c9487f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } } }
I20260924 15:26:38.568385 20938 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:38.568717 20946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:26:38.568727 20939 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef602e27976248f3ba323b4996c9487f. Latest consensus state: current_term: 3 leader_uuid: "ef602e27976248f3ba323b4996c9487f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef602e27976248f3ba323b4996c9487f" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35221 } } }
I20260924 15:26:38.569051 20939 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:38.569684 20946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:26:38.569962 20946 catalog_manager.cc:1259] Loaded cluster ID: c14ada371e5643c098d826ef5c2939cc
I20260924 15:26:38.570163 20946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:26:38.571405 20946 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:26:38.571650 20946 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Loaded TSK: 0
I20260924 15:26:38.571766 20946 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ef602e27976248f3ba323b4996c9487f: Loaded TSK: 1
F20260924 15:26:38.572072 20946 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fecd32619ad google::LogMessage::Fail() at ??:0
@ 0x7fecd326488a google::LogMessage::SendToLog() at ??:0
@ 0x7fecd32614de google::LogMessage::Flush() at ??:0
@ 0x7fecd3262a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fecd59ce104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fecd59cbd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fecdb1bb1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fecdb1bb48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fecdb1bb594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fecdb181c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fecdb18bad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fecd512f0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fecd5124954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fecd5f886db start_thread at ??:0
@ 0x7fecd1d5f61f clone at ??:0
W20260924 15:26:38.671203 20935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:38.671491 20935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:38.671528 20935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:38.673305 20935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:38.673370 20935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:38.673414 20935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:26:38.675655 20935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:32911
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=45499
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:38.675953 20935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:38.676251 20935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:38.681739 20977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:38.681746 20975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:38.681771 20974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:38.682822 20935 server_base.cc:1034] running on GCE node
I20260924 15:26:38.683043 20935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:38.683336 20935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:38.684525 20935 hybrid_clock.cc:648] HybridClock initialized: now 1790263598684507 us; error 27 us; skew 500 ppm
Sep 24 15:26:38 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263598, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:38.687727 20935 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:26:38.688259 20935 webserver.cc:460] Webserver started at http://127.4.78.65:45499/ using document root <none> and password file <none>
I20260924 15:26:38.688501 20935 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:38.688575 20935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:38.690128 20935 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:26:38.690912 20984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:38.691100 20935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:26:38.691212 20935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "2b480a730bbe40c58ff4ecd424b11148"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "14f856c852bb977781b9b3fb2560ec1a"
server_key_iv: "30f582c551cfb5abdb5162c6c0be5382"
server_key_version: "encryptionkey@0"
I20260924 15:26:38.691594 20935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:38.701305 20935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:38.703691 20935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:38.703867 20935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:38.704102 20935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:38.704558 20991 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:26:38.706732 20935 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:26:38.706815 20935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:26:38.706871 20935 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:26:38.707620 20935 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:26:38.707676 20935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:26:38.707754 20991 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Bootstrap starting.
I20260924 15:26:38.715292 20935 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:32911
I20260924 15:26:38.715415 21098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:32911 every 8 connection(s)
I20260924 15:26:38.715785 20935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:26:38.716909 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 20935
I20260924 15:26:38.717211 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:45693
--local_ip_for_outbound_sockets=127.4.78.66
--tserver_master_addrs=127.4.78.126:35221
--webserver_port=41399
--webserver_interface=127.4.78.66
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:26:38.719048 20991 log.cc:824] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:38.722050 20991 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:26:38.722412 20991 tablet_bootstrap.cc:492] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Bootstrap complete.
I20260924 15:26:38.723695 20991 ts_tablet_manager.cc:1397] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260924 15:26:38.725200 20991 raft_consensus.cc:348] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:38.725544 20991 raft_consensus.cc:729] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b480a730bbe40c58ff4ecd424b11148, State: Initialized, Role: FOLLOWER
I20260924 15:26:38.725699 20991 consensus_queue.cc:260] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:38.726003 20991 ts_tablet_manager.cc:1428] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
W20260924 15:26:38.797291 21102 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:26:38.716480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:48573)
0924 15:26:38.716671 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.716717 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:38.717447 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:38.797074 (+ 79627us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
W20260924 15:26:38.797593 21100 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:35221 (0 consecutive failures): Network error: Failed to ping master at 127.4.78.126:35221: Client connection negotiation failed: client connection to 127.4.78.126:35221: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:26:38.798060 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:38.797937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:44727)
0924 15:26:38.797990 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.798030 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:26:38.798421 21102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:38.798333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:35845)
0924 15:26:38.798381 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.798405 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:26:38.798547 21100 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:26:38.806849 21103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:38.807060 21103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:38.807085 21103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:38.808331 21103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:38.808383 21103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:38.808411 21103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:26:38.809983 21103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:45693
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=41399
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:38.810238 21103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:38.810484 21103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:38.815876 21113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:38.815887 21111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:38.815871 21110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:38.816192 21103 server_base.cc:1034] running on GCE node
I20260924 15:26:38.816443 21103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:38.816645 21103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:38.817792 21103 hybrid_clock.cc:648] HybridClock initialized: now 1790263598817777 us; error 27 us; skew 500 ppm
Sep 24 15:26:38 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263598, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:38.820655 21103 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:26:38.821133 21103 webserver.cc:460] Webserver started at http://127.4.78.66:41399/ using document root <none> and password file <none>
I20260924 15:26:38.821391 21103 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:38.821442 21103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:38.822880 21103 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:26:38.823644 21120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:38.823860 21103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:38.823935 21103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "e213ed048adf4b37b0f37feabb02b2cb"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "e6f4f0b2382749bea4caa4015e8a4af4"
server_key_iv: "c606a63c20ae0e13cdfa37ce948234e0"
server_key_version: "encryptionkey@0"
I20260924 15:26:38.824340 21103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:38.842872 21103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:38.845367 21103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:38.845523 21103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:38.845782 21103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:38.846143 21103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:38.846199 21103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:38.846258 21103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:38.846290 21103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:38.852228 21103 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:45693
I20260924 15:26:38.852283 21233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:45693 every 8 connection(s)
I20260924 15:26:38.852636 21103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:26:38.853703 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:38.853335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:32987)
0924 15:26:38.853483 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.853559 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
W20260924 15:26:38.854140 21234 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:35221 (0 consecutive failures): Network error: Failed to ping master at 127.4.78.126:35221: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
I20260924 15:26:38.854414 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:38.854319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:37607)
0924 15:26:38.854368 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.854397 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:26:38.854664 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:38.854613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:40817)
0924 15:26:38.854635 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.854653 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 15:26:38.854770 21234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:26:38.855383 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 21103
I20260924 15:26:38.855671 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:40603
--local_ip_for_outbound_sockets=127.4.78.67
--tserver_master_addrs=127.4.78.126:35221
--webserver_port=45571
--webserver_interface=127.4.78.67
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:34049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:38.938760 21237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:38.938959 21237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:38.938982 21237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:38.940409 21237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:38.940459 21237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:38.940486 21237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:26:38.942085 21237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:34049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:40603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=45571
--tserver_master_addrs=127.4.78.126:35221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:38.942346 21237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:38.942589 21237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:38.947525 21243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:38.947525 21245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:38.947531 21242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:38.947754 21237 server_base.cc:1034] running on GCE node
I20260924 15:26:38.947952 21237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:38.948201 21237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:38.949374 21237 hybrid_clock.cc:648] HybridClock initialized: now 1790263598949369 us; error 38 us; skew 500 ppm
Sep 24 15:26:38 dist-test-slave-l766 krb5kdc[19796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263598, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:38.951908 21237 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:26:38.952353 21237 webserver.cc:460] Webserver started at http://127.4.78.67:45571/ using document root <none> and password file <none>
I20260924 15:26:38.952594 21237 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:38.952663 21237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:38.954015 21237 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:38.954591 21252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:38.954771 21237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:38.954869 21237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "3e460be5cd6f458fbd1f9e5ef13af83c"
format_stamp: "Formatted at 2026-09-24 15:25:57 on dist-test-slave-l766"
server_key: "1c68932bebda7e204eef69f1152ad610"
server_key_iv: "22ddf94d4fa47002b5ad971f43597865"
server_key_version: "encryptionkey@0"
I20260924 15:26:38.955206 21237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:38.968477 21237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:38.970602 21237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:38.970762 21237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:38.970993 21237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:38.971312 21237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:38.971365 21237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:38.971405 21237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:38.971436 21237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:38.977371 21237 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:40603
I20260924 15:26:38.977420 21365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:40603 every 8 connection(s)
I20260924 15:26:38.977746 21237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:26:38.978789 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:38.978430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:43219)
0924 15:26:38.978584 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.978651 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
W20260924 15:26:38.979206 21366 heartbeater.cc:644] Failed to heartbeat to 127.4.78.126:35221 (0 consecutive failures): Network error: Failed to ping master at 127.4.78.126:35221: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
I20260924 15:26:38.979488 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:38.979385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:40031)
0924 15:26:38.979435 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.979459 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:26:38.979792 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:38.979674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:58809)
0924 15:26:38.979721 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:38.979766 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:26:38.979892 21366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:26:38.980697 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 21237
I20260924 15:26:39.799156 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:39.798913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:39627)
0924 15:26:39.799053 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:39.799092 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260924 15:26:39.855381 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:39.855104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:55837)
0924 15:26:39.855267 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:39.855309 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":82,"threads_started":1}
I20260924 15:26:39.943964 21372 raft_consensus.cc:482] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:39.944093 21372 raft_consensus.cc:504] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:39.944753 21372 leader_election.cc:288] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603), e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693)
Sep 24 15:26:39 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263598, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.67@KRBTEST.COM
Sep 24 15:26:39 dist-test-slave-l766 krb5kdc[19796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263598, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.66@KRBTEST.COM
I20260924 15:26:39.955798 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:39.945192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:44575 (local address 127.4.78.66:45693)
0924 15:26:39.945373 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:39.945381 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:39.945408 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:26:39.945889 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:39.945892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:39.945930 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:39.946070 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:39.946785 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:39.947621 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:39.948624 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:39.949010 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:39.949126 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:39.952064 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:39.952072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:39.952072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:39.952081 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:39.953938 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:39.954384 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:39.954386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:39.954386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:39.954482 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:39.954818 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:39.954819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:39.954820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:39.955128 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:39.955220 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:39.955577 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 15:26:39.955717 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":61,"threads_started":1}
I20260924 15:26:39.955878 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:39.945063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:45693 (local address 127.4.78.65:44575)
0924 15:26:39.945167 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:39.945189 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:39.945235 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:39.946090 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:39.946093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:39.946466 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:39.946664 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:39.946670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:39.947723 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:39.947724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:39.948519 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:39.948521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:39.949111 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:39.949112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:39.949237 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:39.949913 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:39.949917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:39.951965 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:39.954011 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:39.954011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:39.954012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:39.954285 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:39.954558 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:39.954559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:39.954559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:39.954743 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:39.955215 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:39.955224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:39.955378 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 15:26:39.955497 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 15:26:39.955774 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":141,"thread_start_us":59,"threads_started":1}
I20260924 15:26:39.956185 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:39.945068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:52465 (local address 127.4.78.67:40603)
0924 15:26:39.945260 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:39.945273 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:39.945352 (+ 79us) server_negotiation.cc:407] Connection header received
0924 15:26:39.945785 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:39.945788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:39.945912 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:39.946049 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:39.946790 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:39.947644 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:39.948611 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:39.949009 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:39.949163 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:39.951705 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:39.951710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:39.951710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:39.951721 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:39.953663 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:39.954304 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:39.954307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:39.954308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:39.954442 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:39.954854 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:39.954855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:39.954855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:39.955128 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:39.955220 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:39.955960 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 15:26:39.956084 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":55,"threads_started":1}
I20260924 15:26:39.956261 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:39.944939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:40603 (local address 127.4.78.65:52465)
0924 15:26:39.945026 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:39.945038 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:39.945102 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:39.946089 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:39.946092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:39.946464 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:39.946664 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:39.946670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:39.947768 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:39.947769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:39.948520 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:39.948522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:39.949117 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:39.949118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:39.949239 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:39.949869 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:39.949879 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:39.951585 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:39.953837 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:39.953844 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:39.953847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:39.954218 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:39.954558 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:39.954559 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:39.954559 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:39.954743 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:39.955215 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:39.955224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:39.955378 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 15:26:39.955497 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 15:26:39.955719 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":68}
W20260924 15:26:39.956913 20988 leader_election.cc:341] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693): Not found: Tablet not found: 7c8b3e681e9040f19b2c312f6cfae246
W20260924 15:26:39.957008 20988 leader_election.cc:341] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603): Not found: Tablet not found: 7c8b3e681e9040f19b2c312f6cfae246
I20260924 15:26:39.957052 20988 leader_election.cc:302] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b480a730bbe40c58ff4ecd424b11148; no voters: 3e460be5cd6f458fbd1f9e5ef13af83c, e213ed048adf4b37b0f37feabb02b2cb
I20260924 15:26:39.957156 21372 raft_consensus.cc:2726] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:26:39.980417 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:39.980199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:47629)
0924 15:26:39.980338 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:39.980381 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260924 15:26:40.799867 21377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:40.799621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:58421)
0924 15:26:40.799779 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:40.799818 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":119,"threads_started":1}
I20260924 15:26:40.856142 21378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:40.855895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:53453)
0924 15:26:40.856057 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:40.856101 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260924 15:26:40.981122 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:40.980892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:37351)
0924 15:26:40.981026 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:40.981066 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260924 15:26:41.528462 21380 raft_consensus.cc:482] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:41.528568 21380 raft_consensus.cc:504] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:41.528757 21380 leader_election.cc:288] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603), e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693)
W20260924 15:26:41.529256 20988 leader_election.cc:341] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603): Not found: Tablet not found: 7c8b3e681e9040f19b2c312f6cfae246
W20260924 15:26:41.529340 20988 leader_election.cc:341] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693): Not found: Tablet not found: 7c8b3e681e9040f19b2c312f6cfae246
I20260924 15:26:41.529357 20988 leader_election.cc:302] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b480a730bbe40c58ff4ecd424b11148; no voters: 3e460be5cd6f458fbd1f9e5ef13af83c, e213ed048adf4b37b0f37feabb02b2cb
I20260924 15:26:41.529433 21380 raft_consensus.cc:2726] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:26:41.800709 21381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:41.800447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:46513)
0924 15:26:41.800596 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:41.800649 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260924 15:26:41.856865 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:41.856619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:57855)
0924 15:26:41.856785 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:41.856822 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
I20260924 15:26:41.981910 21383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:41.981627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:36133)
0924 15:26:41.981744 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:41.981798 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260924 15:26:42.801440 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:42.801197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:60083)
0924 15:26:42.801349 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:42.801393 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
I20260924 15:26:42.857602 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:42.857386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:52873)
0924 15:26:42.857511 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:42.857559 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260924 15:26:42.982591 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:42.982386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:50291)
0924 15:26:42.982516 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:42.982552 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260924 15:26:43.674053 21387 raft_consensus.cc:482] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:43.674266 21387 raft_consensus.cc:504] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:43.674592 21387 leader_election.cc:288] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603), e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693)
W20260924 15:26:43.674758 20988 leader_election.cc:341] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603): Not found: Tablet not found: 7c8b3e681e9040f19b2c312f6cfae246
W20260924 15:26:43.674860 20988 leader_election.cc:341] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693): Not found: Tablet not found: 7c8b3e681e9040f19b2c312f6cfae246
I20260924 15:26:43.674892 20988 leader_election.cc:302] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b480a730bbe40c58ff4ecd424b11148; no voters: 3e460be5cd6f458fbd1f9e5ef13af83c, e213ed048adf4b37b0f37feabb02b2cb
I20260924 15:26:43.674984 21387 raft_consensus.cc:2726] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:26:43.802218 21388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:43.802002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:40995)
0924 15:26:43.802132 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:43.802173 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260924 15:26:43.858453 21389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:43.858196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:35467)
0924 15:26:43.858372 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:43.858410 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":103,"threads_started":1}
I20260924 15:26:43.983271 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:43.983043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:51249)
0924 15:26:43.983178 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:43.983217 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260924 15:26:44.802935 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:44.802709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:56675)
0924 15:26:44.802849 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:44.802888 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260924 15:26:44.859083 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:44.858860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:57947)
0924 15:26:44.859007 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:44.859042 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":59,"threads_started":1}
I20260924 15:26:44.983964 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:44.983727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:53467)
0924 15:26:44.983873 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:44.983915 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260924 15:26:45.803644 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:45.803392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:34511)
0924 15:26:45.803522 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:45.803566 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260924 15:26:45.859731 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:45.859533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:34429)
0924 15:26:45.859659 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:45.859692 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260924 15:26:45.984627 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:45.984404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:38817)
0924 15:26:45.984543 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:45.984578 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260924 15:26:46.804308 21397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:46.804044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:39187)
0924 15:26:46.804200 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:46.804262 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260924 15:26:46.860366 21398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:46.860164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:37281)
0924 15:26:46.860289 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:46.860325 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260924 15:26:46.985351 21399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:46.985114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:58387)
0924 15:26:46.985270 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:46.985309 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":63,"threads_started":1}
I20260924 15:26:47.805086 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:47.804815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:38335)
0924 15:26:47.804993 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:47.805036 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260924 15:26:47.861111 21401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:47.860888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:56419)
0924 15:26:47.861025 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:47.861063 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260924 15:26:47.986030 21402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:47.985798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:42811)
0924 15:26:47.985949 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:47.985985 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260924 15:26:48.417982 21403 raft_consensus.cc:482] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:48.418097 21403 raft_consensus.cc:504] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e460be5cd6f458fbd1f9e5ef13af83c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 40603 } } peers { permanent_uuid: "e213ed048adf4b37b0f37feabb02b2cb" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 45693 } } peers { permanent_uuid: "2b480a730bbe40c58ff4ecd424b11148" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 32911 } }
I20260924 15:26:48.418282 21403 leader_election.cc:288] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603), e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693)
W20260924 15:26:48.418753 20988 leader_election.cc:341] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e213ed048adf4b37b0f37feabb02b2cb (127.4.78.66:45693): Not found: Tablet not found: 7c8b3e681e9040f19b2c312f6cfae246
W20260924 15:26:48.418860 20988 leader_election.cc:341] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3e460be5cd6f458fbd1f9e5ef13af83c (127.4.78.67:40603): Not found: Tablet not found: 7c8b3e681e9040f19b2c312f6cfae246
I20260924 15:26:48.418919 20988 leader_election.cc:302] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b480a730bbe40c58ff4ecd424b11148; no voters: 3e460be5cd6f458fbd1f9e5ef13af83c, e213ed048adf4b37b0f37feabb02b2cb
I20260924 15:26:48.419031 21403 raft_consensus.cc:2726] T 7c8b3e681e9040f19b2c312f6cfae246 P 2b480a730bbe40c58ff4ecd424b11148 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:26:48.805889 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:48.805607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:53733)
0924 15:26:48.805750 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:48.805792 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260924 15:26:48.861840 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:48.861584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:42335)
0924 15:26:48.861712 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:48.861741 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260924 15:26:48.986730 21406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:48.986515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:42411)
0924 15:26:48.986651 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:48.986685 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260924 15:26:49.806743 21407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:49.806428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:48409)
0924 15:26:49.806610 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:49.806656 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":96,"threads_started":1}
I20260924 15:26:49.862565 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:49.862337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:56751)
0924 15:26:49.862474 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:49.862517 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260924 15:26:49.987432 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:49.987192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:46473)
0924 15:26:49.987339 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:49.987381 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260924 15:26:50.807513 21410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:50.807265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:54751)
0924 15:26:50.807426 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:50.807465 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":106,"threads_started":1}
I20260924 15:26:50.863255 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:50.862982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:47865)
0924 15:26:50.863129 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:50.863208 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260924 15:26:50.988315 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:50.988036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:56229)
0924 15:26:50.988180 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:50.988259 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260924 15:26:51.808282 21413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:51.808034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:47499)
0924 15:26:51.808166 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:51.808202 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260924 15:26:51.863940 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:51.863711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:53437)
0924 15:26:51.863858 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:51.863897 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":61,"threads_started":1}
I20260924 15:26:51.988912 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:51.988717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:56897)
0924 15:26:51.988831 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:51.988867 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260924 15:26:52.809010 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:52.808777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:54039)
0924 15:26:52.808921 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:52.808962 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260924 15:26:52.864626 21417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:52.864414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:37471)
0924 15:26:52.864541 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:52.864580 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260924 15:26:52.989669 21418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:52.989456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:38933)
0924 15:26:52.989587 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:52.989624 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260924 15:26:53.809893 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:53.809621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.65:38357)
0924 15:26:53.809761 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:53.809794 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260924 15:26:53.865379 21420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:53.865130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.66:42125)
0924 15:26:53.865296 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:53.865336 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
I20260924 15:26:53.983556 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 20935
I20260924 15:26:53.988740 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 21103
I20260924 15:26:53.990613 21421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:53.990355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35221 (local address 127.4.78.67:42613)
0924 15:26:53.990528 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:53.990569 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.126:35221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":75,"threads_started":1}
I20260924 15:26:53.993683 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 21237
2026-09-24T15:26:53Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58372 ms)
[----------] 30 tests from SecurityITest (231640 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:26:54 dist-test-slave-l766 krb5kdc[21424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:26:54 dist-test-slave-l766 krb5kdc[21424](info): set up 2 sockets
Sep 24 15:26:54 dist-test-slave-l766 krb5kdc[21424](info): commencing operation
krb5kdc: starting...
W20260924 15:26:55.044624 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:26:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:26:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:26:55.162935 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:40071
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:40071
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:55.245541 21437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:55.245733 21437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:55.245759 21437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:55.246933 21437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:26:55.246978 21437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:55.246994 21437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:55.247006 21437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:26:55.247057 21437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:26:55.248481 21437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:40071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:40071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:55.248721 21437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:55.248946 21437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:55.253615 21442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:55.253647 21443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:55.253645 21445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:55.253885 21437 server_base.cc:1034] running on GCE node
I20260924 15:26:55.254149 21437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:55.254415 21437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:55.255573 21437 hybrid_clock.cc:648] HybridClock initialized: now 1790263615255551 us; error 33 us; skew 500 ppm
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:55.261504 21437 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:26:55.261958 21437 webserver.cc:460] Webserver started at http://127.4.78.126:36189/ using document root <none> and password file <none>
I20260924 15:26:55.262141 21437 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:55.262192 21437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:55.262284 21437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:55.263290 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "3697177433f14e83afe2eb3e558c6019"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "c9a4669773e67f62f901191994d1f173"
server_key_iv: "f9c611471a2cf1544518fd90fb2c595e"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.263588 21437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "3697177433f14e83afe2eb3e558c6019"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "c9a4669773e67f62f901191994d1f173"
server_key_iv: "f9c611471a2cf1544518fd90fb2c595e"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.264865 21437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:26:55.265596 21452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.265769 21437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:55.265879 21437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "3697177433f14e83afe2eb3e558c6019"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "c9a4669773e67f62f901191994d1f173"
server_key_iv: "f9c611471a2cf1544518fd90fb2c595e"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.265937 21437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:55.279960 21437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:55.282455 21437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:55.282573 21437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:55.286334 21437 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:40071
I20260924 15:26:55.286388 21505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:40071 every 8 connection(s)
I20260924 15:26:55.286742 21437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:26:55.287505 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 21437
I20260924 15:26:55.287612 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:26:55.287829 21506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:55.287846 4409 external_mini_cluster.cc:1428] Setting key e38e4cbd59cc5548d32b3333befbdb59
I20260924 15:26:55.291249 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019: Bootstrap starting.
I20260924 15:26:55.291924 21506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:55.292230 21506 log.cc:824] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:55.292893 21506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019: No bootstrap required, opened a new log
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:55.294836 21506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3697177433f14e83afe2eb3e558c6019" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40071 } }
I20260924 15:26:55.295010 21506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:55.295073 21506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3697177433f14e83afe2eb3e558c6019, State: Initialized, Role: FOLLOWER
I20260924 15:26:55.295207 21506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3697177433f14e83afe2eb3e558c6019" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40071 } }
I20260924 15:26:55.295308 21506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:26:55.295353 21506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:26:55.295406 21506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:55.296093 21506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3697177433f14e83afe2eb3e558c6019" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40071 } }
I20260924 15:26:55.296206 21506 leader_election.cc:302] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3697177433f14e83afe2eb3e558c6019; no voters:
I20260924 15:26:55.296376 21506 leader_election.cc:288] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:26:55.296443 21511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:55.296617 21511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [term 1 LEADER]: Becoming Leader. State: Replica: 3697177433f14e83afe2eb3e558c6019, State: Running, Role: LEADER
I20260924 15:26:55.296630 21506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:26:55.296777 21511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3697177433f14e83afe2eb3e558c6019" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40071 } }
I20260924 15:26:55.297163 21511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3697177433f14e83afe2eb3e558c6019. Latest consensus state: current_term: 1 leader_uuid: "3697177433f14e83afe2eb3e558c6019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3697177433f14e83afe2eb3e558c6019" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40071 } } }
I20260924 15:26:55.297415 21511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:55.297338 21512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3697177433f14e83afe2eb3e558c6019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3697177433f14e83afe2eb3e558c6019" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40071 } } }
I20260924 15:26:55.297865 21512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019 [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:55.298036 21520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:26:55.298923 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.288410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55912 (local address 127.4.78.126:40071)
0924 15:26:55.288590 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.288598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.288624 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:26:55.289029 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.289035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.289081 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.289207 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:55.289980 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.290874 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.291927 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.292407 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.292528 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.294727 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:55.294732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:55.294733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:55.294751 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:55.296857 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.297287 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.297290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.297291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.297403 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.297628 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.297629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.297629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.298006 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:55.298569 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.298656 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.298781 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260924 15:26:55.299045 21520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:26:55.300536 21520 catalog_manager.cc:1347] Generated new cluster ID: bca95b075e0e4785b5aa0074a9970d36
I20260924 15:26:55.300586 21520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:26:55.311378 21520 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:26:55.312063 21520 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:26:55.316378 21520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019: Generated new TSK 0
I20260924 15:26:55.316558 21520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:26:55.353273 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40071
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:55.434986 21532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:55.435195 21532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:55.435217 21532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:55.436375 21532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:55.436427 21532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:55.436455 21532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:26:55.437974 21532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:55.438206 21532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:55.438454 21532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:55.443717 21537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:55.443708 21538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:55.443737 21540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:55.443994 21532 server_base.cc:1034] running on GCE node
I20260924 15:26:55.444216 21532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:55.444545 21532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:55.445756 21532 hybrid_clock.cc:648] HybridClock initialized: now 1790263615445726 us; error 48 us; skew 500 ppm
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:55.448206 21532 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:26:55.448655 21532 webserver.cc:460] Webserver started at http://127.4.78.65:42271/ using document root <none> and password file <none>
I20260924 15:26:55.448879 21532 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:55.448946 21532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:55.449097 21532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:55.450203 21532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "22543368468f4e4990e724a891c8e171"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "795926eda299e850b8ca88c55968f5e1"
server_key_iv: "5127cdf74af24051c66f1f2f3c63b3bb"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.450569 21532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "22543368468f4e4990e724a891c8e171"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "795926eda299e850b8ca88c55968f5e1"
server_key_iv: "5127cdf74af24051c66f1f2f3c63b3bb"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.451853 21532 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:26:55.452560 21547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.452721 21532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:55.452823 21532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "22543368468f4e4990e724a891c8e171"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "795926eda299e850b8ca88c55968f5e1"
server_key_iv: "5127cdf74af24051c66f1f2f3c63b3bb"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.452906 21532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:55.472069 21532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:55.473943 21532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:55.474105 21532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:55.474337 21532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:55.474637 21532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:55.474690 21532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.474730 21532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:55.474758 21532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.480541 21532 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:36167
I20260924 15:26:55.480594 21660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:36167 every 8 connection(s)
I20260924 15:26:55.480890 21532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:55.488176 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 21532
I20260924 15:26:55.488292 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:26:55.488502 4409 external_mini_cluster.cc:1428] Setting key 53730cc788b3c27a92e0a2ef7342dfcb
I20260924 15:26:55.490283 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.481579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:36631 (local address 127.4.78.126:40071)
0924 15:26:55.481643 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.481645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.481870 (+ 225us) server_negotiation.cc:407] Connection header received
0924 15:26:55.482443 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.482444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.482456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.482522 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:55.483485 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.483977 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.484948 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.485140 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.485194 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.487368 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:55.487369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:55.487370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:55.487377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:55.488827 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.489312 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.489313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.489314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.489369 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.489654 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.489655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.489655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.489749 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:55.489795 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.490151 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.490220 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:26:55.490574 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.481526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40071 (local address 127.4.78.65:36631)
0924 15:26:55.481685 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:55.481726 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:55.482311 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:55.482600 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:55.482604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:55.483171 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:55.483380 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.483385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.484058 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.484058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.484873 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.484875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.485212 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.485212 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.485349 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.485888 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:55.485898 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:55.487228 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:55.488904 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:55.488911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:55.488913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:55.489237 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:55.489446 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:55.489446 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:55.489447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:55.489589 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:55.489831 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:55.489837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:55.490009 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 15:26:55.490128 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 15:26:55.490301 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260924 15:26:55.491148 21661 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40071
I20260924 15:26:55.491259 21661 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:55.491443 21661 heartbeater.cc:511] Master 127.4.78.126:40071 requested a full tablet report, sending...
I20260924 15:26:55.491935 21469 ts_manager.cc:194] Registered new tserver with Master: 22543368468f4e4990e724a891c8e171 (127.4.78.65:36167)
I20260924 15:26:55.492532 21469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:36631
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:26:55.512753 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40071
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:55.592826 21666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:55.593073 21666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:55.593111 21666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:55.594323 21666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:55.594393 21666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:55.594440 21666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:26:55.595908 21666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:55.596170 21666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:55.596398 21666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:55.601536 21672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:55.601549 21674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:55.601567 21671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:55.601755 21666 server_base.cc:1034] running on GCE node
I20260924 15:26:55.601964 21666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:55.602161 21666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:55.603297 21666 hybrid_clock.cc:648] HybridClock initialized: now 1790263615603279 us; error 41 us; skew 500 ppm
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:55.605619 21666 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:26:55.606050 21666 webserver.cc:460] Webserver started at http://127.4.78.66:38475/ using document root <none> and password file <none>
I20260924 15:26:55.606252 21666 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:55.606293 21666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:55.606383 21666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:55.607416 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "c9ca2d00591a40119173b6d56ba4ebd8"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "76620a5843cf9744cf1bc97b4e12e5c6"
server_key_iv: "39a8e469b9193eec848cd58b50756c90"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.607698 21666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "c9ca2d00591a40119173b6d56ba4ebd8"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "76620a5843cf9744cf1bc97b4e12e5c6"
server_key_iv: "39a8e469b9193eec848cd58b50756c90"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.608889 21666 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:26:55.609563 21681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.609717 21666 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:55.609797 21666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "c9ca2d00591a40119173b6d56ba4ebd8"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "76620a5843cf9744cf1bc97b4e12e5c6"
server_key_iv: "39a8e469b9193eec848cd58b50756c90"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.609905 21666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:55.619910 21666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:55.621592 21666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:55.621697 21666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:55.621989 21666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:55.622290 21666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:55.622323 21666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.622344 21666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:55.622395 21666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.628099 21666 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:34037
I20260924 15:26:55.628310 21794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:34037 every 8 connection(s)
I20260924 15:26:55.628511 21666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:55.637120 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.629297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:56331 (local address 127.4.78.126:40071)
0924 15:26:55.629362 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.629365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.629526 (+ 161us) server_negotiation.cc:407] Connection header received
0924 15:26:55.630084 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.630085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.630094 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.630154 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:55.631062 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.631550 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.632495 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.632673 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.632722 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.634790 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:55.634791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:55.634792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:55.634797 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:55.635994 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.636345 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.636346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.636346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.636395 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.636592 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.636592 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.636593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.636689 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:55.636737 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.636992 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.637058 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:26:55.637236 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.629245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40071 (local address 127.4.78.66:56331)
0924 15:26:55.629400 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:55.629441 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:55.629992 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:55.630229 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:55.630233 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:55.630785 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:55.630979 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.630985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.631626 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.631627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.632419 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.632420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.632742 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.632743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.632843 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.633377 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:55.633385 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:55.634705 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:55.636066 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:55.636072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:55.636073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:55.636275 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:55.636449 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:55.636449 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:55.636449 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:55.636539 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:55.636743 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:55.636747 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:55.636860 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 15:26:55.636941 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:26:55.637053 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260924 15:26:55.637745 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 21666
I20260924 15:26:55.637777 21795 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40071
I20260924 15:26:55.637888 21795 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:55.637887 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:26:55.638088 21795 heartbeater.cc:511] Master 127.4.78.126:40071 requested a full tablet report, sending...
I20260924 15:26:55.638118 4409 external_mini_cluster.cc:1428] Setting key 5c48207269e5bd6ee531e3516438cfec
I20260924 15:26:55.638540 21468 ts_manager.cc:194] Registered new tserver with Master: c9ca2d00591a40119173b6d56ba4ebd8 (127.4.78.66:34037)
I20260924 15:26:55.638931 21468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:56331
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:26:55.664338 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40071
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:55.747855 21800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:55.748083 21800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:55.748108 21800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:55.749402 21800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:55.749449 21800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:55.749477 21800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:26:55.751108 21800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40071
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:55.751367 21800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:55.751639 21800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:26:55.756825 21800 server_base.cc:1034] running on GCE node
W20260924 15:26:55.756808 21806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:55.756776 21805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:55.756788 21808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:55.757175 21800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:55.757468 21800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:55.758649 21800 hybrid_clock.cc:648] HybridClock initialized: now 1790263615758633 us; error 43 us; skew 500 ppm
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:55.761207 21800 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:26:55.761718 21800 webserver.cc:460] Webserver started at http://127.4.78.67:35599/ using document root <none> and password file <none>
I20260924 15:26:55.761993 21800 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:55.762070 21800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:55.762251 21800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:55.763447 21800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "4c607f042c8248a6ac74c8c91fcacc9c"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "cb6f8586adbdf1f6e07a19f02bca5923"
server_key_iv: "7ff4f4dadc1ce6620c049624dd044a8b"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.763849 21800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "4c607f042c8248a6ac74c8c91fcacc9c"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "cb6f8586adbdf1f6e07a19f02bca5923"
server_key_iv: "7ff4f4dadc1ce6620c049624dd044a8b"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.765288 21800 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:26:55.766171 21815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.766387 21800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:55.766508 21800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "4c607f042c8248a6ac74c8c91fcacc9c"
format_stamp: "Formatted at 2026-09-24 15:26:55 on dist-test-slave-l766"
server_key: "cb6f8586adbdf1f6e07a19f02bca5923"
server_key_iv: "7ff4f4dadc1ce6620c049624dd044a8b"
server_key_version: "encryptionkey@0"
I20260924 15:26:55.766587 21800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:55.774453 21800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:55.776470 21800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:55.776643 21800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:55.776872 21800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:55.777233 21800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:55.777292 21800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.777336 21800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:55.777369 21800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:55.783809 21800 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:35581
I20260924 15:26:55.783862 21928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:35581 every 8 connection(s)
I20260924 15:26:55.784178 21800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:26:55.789352 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 21800
I20260924 15:26:55.789474 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:26:55.789700 4409 external_mini_cluster.cc:1428] Setting key e145afac8797dbdcca5033da01e07309
Sep 24 15:26:55 dist-test-slave-l766 krb5kdc[21424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263615, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:55.793931 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.784859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:46087 (local address 127.4.78.126:40071)
0924 15:26:55.784936 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.784938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.785089 (+ 151us) server_negotiation.cc:407] Connection header received
0924 15:26:55.785554 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.785555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.785565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.785628 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:55.786656 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.787170 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.788173 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.788368 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.788421 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.791055 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:55.791056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:55.791057 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:55.791063 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:55.792621 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.792995 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.792996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.792996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.793052 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.793264 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.793265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.793265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.793371 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:55.793422 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.793652 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.793782 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:26:55.794049 21931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.784793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40071 (local address 127.4.78.67:46087)
0924 15:26:55.784955 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:55.784995 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:55.785464 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:55.785706 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:55.785710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:55.786350 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:55.786541 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.786546 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.787261 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.787262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.788088 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.788091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.788441 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.788442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.788540 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.789023 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:55.789031 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:55.790975 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:55.792696 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:55.792702 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:55.792703 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:55.792912 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:55.793114 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:55.793115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:55.793115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:55.793206 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:55.793426 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:55.793430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:55.793544 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 15:26:55.793625 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:26:55.793781 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260924 15:26:55.794785 21929 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40071
I20260924 15:26:55.794906 21929 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:55.795128 21929 heartbeater.cc:511] Master 127.4.78.126:40071 requested a full tablet report, sending...
I20260924 15:26:55.795477 21468 ts_manager.cc:194] Registered new tserver with Master: 4c607f042c8248a6ac74c8c91fcacc9c (127.4.78.67:35581)
I20260924 15:26:55.795864 21468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:46087
I20260924 15:26:55.801620 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:26:55.809227 21509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.802943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55920 (local address 127.4.78.126:40071)
0924 15:26:55.803018 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.803021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.803110 (+ 89us) server_negotiation.cc:407] Connection header received
0924 15:26:55.803187 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.803188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.803200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.803264 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:55.803831 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.804289 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.805023 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.805212 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.805298 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.806521 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:55.806523 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:55.806523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:55.806530 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:55.807902 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.808379 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.808380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.808380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.808438 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:55.808637 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:55.808638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:55.808638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:55.808788 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:55.808893 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.809034 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.809114 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 15:26:55.810948 21468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:26:55.812625 21468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:26:55.819861 21938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.814957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36167 (local address 127.0.0.1:45708)
0924 15:26:55.815083 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:55.815126 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:55.815193 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:55.816140 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:55.816141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:55.816143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:55.816205 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.816206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.817665 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.817666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.818842 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.818844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.819608 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.819608 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.819685 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.819689 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:55.819747 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:26:55.819790 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260924 15:26:55.819909 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.815322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45708 (local address 127.4.78.65:36167)
0924 15:26:55.815968 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.815974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.815984 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:26:55.816023 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.816025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.816040 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.816292 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:55.816370 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.817517 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.818935 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.819522 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.819604 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.819680 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.819764 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.819815 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":102,"threads_started":1}
I20260924 15:26:55.819968 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.815049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.4.78.67:35581)
0924 15:26:55.815341 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.815346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.815357 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:26:55.815396 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.815398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.815414 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.815491 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:55.815771 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.816949 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.818737 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.819407 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.819465 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.819510 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.819832 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.819875 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":95,"threads_started":1}
I20260924 15:26:55.820091 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.814960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:35581 (local address 127.0.0.1:48678)
0924 15:26:55.815102 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:55.815126 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:55.815193 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:55.815591 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:55.815594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:55.815596 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:55.815667 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.815669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.817094 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.817095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.818631 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.818634 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.819548 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.819549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.819620 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.819624 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:55.819684 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:26:55.819747 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":32,"thread_start_us":55,"threads_started":1}
I20260924 15:26:55.820376 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.815574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59184 (local address 127.4.78.66:34037)
0924 15:26:55.815846 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.815852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.815865 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:26:55.815910 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.815912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.815930 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.816190 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:55.816264 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.817038 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.818582 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.819355 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.819441 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.819509 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.820250 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.820300 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260924 15:26:55.820609 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.815159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:34037 (local address 127.0.0.1:59184)
0924 15:26:55.815760 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:55.815770 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:55.815815 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:55.816043 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:55.816045 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:55.816047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:55.816111 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.816113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.817202 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.817203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.818465 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.818468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.819459 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.819460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.819538 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.819555 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 15:26:55.819622 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:26:55.819710 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"mutex_wait_us":69,"thread_start_us":36,"threads_started":1}
I20260924 15:26:55.820874 21863 tablet_service.cc:1467] Processing CreateTablet for tablet 10542ab54a8d4f4ca80634e48aaa8433 (DEFAULT_TABLE table=test-table [id=f152bf79c1e14500a915cce26e0f63d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:55.820958 21595 tablet_service.cc:1467] Processing CreateTablet for tablet 10542ab54a8d4f4ca80634e48aaa8433 (DEFAULT_TABLE table=test-table [id=f152bf79c1e14500a915cce26e0f63d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:55.821288 21863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10542ab54a8d4f4ca80634e48aaa8433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:55.821301 21595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10542ab54a8d4f4ca80634e48aaa8433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:55.821305 21729 tablet_service.cc:1467] Processing CreateTablet for tablet 10542ab54a8d4f4ca80634e48aaa8433 (DEFAULT_TABLE table=test-table [id=f152bf79c1e14500a915cce26e0f63d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:55.821601 21729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10542ab54a8d4f4ca80634e48aaa8433. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:55.824255 21943 tablet_bootstrap.cc:492] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: Bootstrap starting.
I20260924 15:26:55.824805 21944 tablet_bootstrap.cc:492] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171: Bootstrap starting.
I20260924 15:26:55.825027 21943 tablet_bootstrap.cc:654] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:55.825526 21944 tablet_bootstrap.cc:654] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:55.825664 21943 log.cc:824] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:55.825981 21945 tablet_bootstrap.cc:492] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Bootstrap starting.
I20260924 15:26:55.826179 21944 log.cc:824] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:55.826748 21945 tablet_bootstrap.cc:654] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:55.826983 21943 tablet_bootstrap.cc:492] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: No bootstrap required, opened a new log
I20260924 15:26:55.827061 21943 ts_tablet_manager.cc:1397] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:26:55.827462 21945 log.cc:824] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:55.828331 21944 tablet_bootstrap.cc:492] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171: No bootstrap required, opened a new log
I20260924 15:26:55.828415 21944 ts_tablet_manager.cc:1397] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 15:26:55.828761 21945 tablet_bootstrap.cc:492] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: No bootstrap required, opened a new log
I20260924 15:26:55.828819 21945 ts_tablet_manager.cc:1397] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:26:55.829274 21943 raft_consensus.cc:348] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.829478 21943 raft_consensus.cc:374] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:55.829520 21943 raft_consensus.cc:729] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c607f042c8248a6ac74c8c91fcacc9c, State: Initialized, Role: FOLLOWER
I20260924 15:26:55.829684 21943 consensus_queue.cc:260] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.830014 21943 ts_tablet_manager.cc:1428] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:26:55.830152 21929 heartbeater.cc:503] Master 127.4.78.126:40071 was elected leader, sending a full tablet report...
I20260924 15:26:55.830729 21944 raft_consensus.cc:348] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.830919 21944 raft_consensus.cc:374] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:55.830957 21944 raft_consensus.cc:729] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22543368468f4e4990e724a891c8e171, State: Initialized, Role: FOLLOWER
I20260924 15:26:55.831060 21944 consensus_queue.cc:260] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.831357 21944 ts_tablet_manager.cc:1428] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:26:55.831383 21661 heartbeater.cc:503] Master 127.4.78.126:40071 was elected leader, sending a full tablet report...
I20260924 15:26:55.831600 21945 raft_consensus.cc:348] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.831765 21945 raft_consensus.cc:374] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:55.831813 21945 raft_consensus.cc:729] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9ca2d00591a40119173b6d56ba4ebd8, State: Initialized, Role: FOLLOWER
I20260924 15:26:55.831923 21945 consensus_queue.cc:260] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.832195 21945 ts_tablet_manager.cc:1428] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:26:55.832201 21795 heartbeater.cc:503] Master 127.4.78.126:40071 was elected leader, sending a full tablet report...
W20260924 15:26:55.879124 21796 tablet.cc:2369] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:26:55.970459 21951 raft_consensus.cc:482] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:55.970582 21951 raft_consensus.cc:504] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.970908 21951 leader_election.cc:288] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c607f042c8248a6ac74c8c91fcacc9c (127.4.78.67:35581), 22543368468f4e4990e724a891c8e171 (127.4.78.65:36167)
I20260924 15:26:55.973904 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.971078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:35581 (local address 127.4.78.66:36329)
0924 15:26:55.971127 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:55.971142 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:55.971199 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:55.971395 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:55.971396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:55.971398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:55.971499 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.971501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.972213 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.972214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.972984 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.972987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.973623 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.973624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.973712 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.973716 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:55.973757 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 15:26:55.973789 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":1}
I20260924 15:26:55.973929 21939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.971220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:36329 (local address 127.4.78.67:35581)
0924 15:26:55.971263 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.971265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.971273 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:26:55.971307 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.971307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.971316 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.971388 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:55.971600 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.972134 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.973087 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.973538 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.973593 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.973602 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.973787 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.973856 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:26:55.973975 21941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.971175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:35811 (local address 127.4.78.65:36167)
0924 15:26:55.971230 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.971232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.971470 (+ 238us) server_negotiation.cc:407] Connection header received
0924 15:26:55.971500 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.971501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.971515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.971572 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:55.971750 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.972239 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.973206 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.973564 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.973656 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.973665 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.973861 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.973899 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:26:55.974042 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.971078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36167 (local address 127.4.78.66:35811)
0924 15:26:55.971396 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:55.971405 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:55.971460 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:55.971597 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:55.971598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:55.971600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:55.971683 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.971684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.972332 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.972333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.973121 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:55.973124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.973681 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.973682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:55.973759 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.973764 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:26:55.973854 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 15:26:55.973890 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":120,"thread_start_us":126,"threads_started":1}
I20260924 15:26:55.974299 21883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10542ab54a8d4f4ca80634e48aaa8433" candidate_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" is_pre_election: true
I20260924 15:26:55.974469 21883 raft_consensus.cc:2454] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ca2d00591a40119173b6d56ba4ebd8 in term 0.
I20260924 15:26:55.974501 21615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10542ab54a8d4f4ca80634e48aaa8433" candidate_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22543368468f4e4990e724a891c8e171" is_pre_election: true
I20260924 15:26:55.974676 21615 raft_consensus.cc:2454] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9ca2d00591a40119173b6d56ba4ebd8 in term 0.
I20260924 15:26:55.974700 21685 leader_election.cc:302] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c607f042c8248a6ac74c8c91fcacc9c, c9ca2d00591a40119173b6d56ba4ebd8; no voters:
I20260924 15:26:55.974841 21951 raft_consensus.cc:2781] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:26:55.974889 21951 raft_consensus.cc:482] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:26:55.974911 21951 raft_consensus.cc:3037] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:55.975644 21951 raft_consensus.cc:504] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.975843 21951 leader_election.cc:288] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [CANDIDATE]: Term 1 election: Requested vote from peers 4c607f042c8248a6ac74c8c91fcacc9c (127.4.78.67:35581), 22543368468f4e4990e724a891c8e171 (127.4.78.65:36167)
I20260924 15:26:55.976082 21883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10542ab54a8d4f4ca80634e48aaa8433" candidate_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c607f042c8248a6ac74c8c91fcacc9c"
I20260924 15:26:55.976179 21615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10542ab54a8d4f4ca80634e48aaa8433" candidate_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22543368468f4e4990e724a891c8e171"
I20260924 15:26:55.976271 21615 raft_consensus.cc:3037] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:55.976173 21883 raft_consensus.cc:3037] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:55.976996 21615 raft_consensus.cc:2454] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ca2d00591a40119173b6d56ba4ebd8 in term 1.
I20260924 15:26:55.976996 21883 raft_consensus.cc:2454] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9ca2d00591a40119173b6d56ba4ebd8 in term 1.
I20260924 15:26:55.977211 21682 leader_election.cc:302] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22543368468f4e4990e724a891c8e171, c9ca2d00591a40119173b6d56ba4ebd8; no voters:
I20260924 15:26:55.977324 21951 raft_consensus.cc:2781] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:55.977466 21951 raft_consensus.cc:686] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 1 LEADER]: Becoming Leader. State: Replica: c9ca2d00591a40119173b6d56ba4ebd8, State: Running, Role: LEADER
I20260924 15:26:55.977566 21951 consensus_queue.cc:237] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } }
I20260924 15:26:55.978474 21468 catalog_manager.cc:5495] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c9ca2d00591a40119173b6d56ba4ebd8 (127.4.78.66). New cstate: current_term: 1 leader_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ca2d00591a40119173b6d56ba4ebd8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 34037 } health_report { overall_health: HEALTHY } } }
W20260924 15:26:55.981554 21662 tablet.cc:2369] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:26:55.999197 21942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:55.996573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.4.78.66:34037)
0924 15:26:55.996642 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:55.996644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:55.996653 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:26:55.996702 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:55.996703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:55.996728 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:55.996822 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:26:55.996976 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.997477 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.998420 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:55.998622 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:55.998697 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:55.998868 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:26:55.998977 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:26:55.999075 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:55.999094 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 15:26:55.999114 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 15:26:56.000993 21883 raft_consensus.cc:1261] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 1 FOLLOWER]: Refusing update from remote peer c9ca2d00591a40119173b6d56ba4ebd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:56.001001 21615 raft_consensus.cc:1261] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 1 FOLLOWER]: Refusing update from remote peer c9ca2d00591a40119173b6d56ba4ebd8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:56.001344 21953 consensus_queue.cc:1035] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c607f042c8248a6ac74c8c91fcacc9c" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 35581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:56.001513 21951 consensus_queue.cc:1035] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22543368468f4e4990e724a891c8e171" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 36167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:56.006021 21468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:56.006124 21468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:56.006881 21468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3697177433f14e83afe2eb3e558c6019: Sending DeleteTablet for 3 replicas of tablet 10542ab54a8d4f4ca80634e48aaa8433
I20260924 15:26:56.007181 21863 tablet_service.cc:1514] Processing DeleteTablet for tablet 10542ab54a8d4f4ca80634e48aaa8433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:56 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:48678
I20260924 15:26:56.007223 21729 tablet_service.cc:1514] Processing DeleteTablet for tablet 10542ab54a8d4f4ca80634e48aaa8433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:56 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:59184
I20260924 15:26:56.007221 21595 tablet_service.cc:1514] Processing DeleteTablet for tablet 10542ab54a8d4f4ca80634e48aaa8433 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:56 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:45708
I20260924 15:26:56.007473 21968 tablet_replica.cc:330] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171: stopping tablet replica
I20260924 15:26:56.007498 21966 tablet_replica.cc:330] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: stopping tablet replica
I20260924 15:26:56.007579 21968 raft_consensus.cc:2229] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:56.007678 21968 raft_consensus.cc:2258] T 10542ab54a8d4f4ca80634e48aaa8433 P 22543368468f4e4990e724a891c8e171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:56.007792 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 21532
I20260924 15:26:56.007746 21966 raft_consensus.cc:2229] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:56.008400 21966 raft_consensus.cc:2258] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:56.008648 21967 tablet_replica.cc:330] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: stopping tablet replica
I20260924 15:26:56.008736 21967 raft_consensus.cc:2229] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:26:56.008828 21967 raft_consensus.cc:2258] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:56.009027 21966 ts_tablet_manager.cc:1905] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:56.009249 21967 ts_tablet_manager.cc:1905] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:56.010797 21966 ts_tablet_manager.cc:1918] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:56.010927 21966 log.cc:1196] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/10542ab54a8d4f4ca80634e48aaa8433
I20260924 15:26:56.011277 21966 ts_tablet_manager.cc:1939] T 10542ab54a8d4f4ca80634e48aaa8433 P 4c607f042c8248a6ac74c8c91fcacc9c: Deleting consensus metadata
I20260924 15:26:56.011351 21967 ts_tablet_manager.cc:1918] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:56.011444 21967 log.cc:1196] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/10542ab54a8d4f4ca80634e48aaa8433
I20260924 15:26:56.011737 21967 ts_tablet_manager.cc:1939] T 10542ab54a8d4f4ca80634e48aaa8433 P c9ca2d00591a40119173b6d56ba4ebd8: Deleting consensus metadata
I20260924 15:26:56.011823 21456 catalog_manager.cc:4841] TS 4c607f042c8248a6ac74c8c91fcacc9c (127.4.78.67:35581): tablet 10542ab54a8d4f4ca80634e48aaa8433 (table test-table [id=f152bf79c1e14500a915cce26e0f63d4]) successfully deleted
I20260924 15:26:56.012264 21456 catalog_manager.cc:4841] TS c9ca2d00591a40119173b6d56ba4ebd8 (127.4.78.66:34037): tablet 10542ab54a8d4f4ca80634e48aaa8433 (table test-table [id=f152bf79c1e14500a915cce26e0f63d4]) successfully deleted
W20260924 15:26:56.015265 21453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:36167 (error 108)
I20260924 15:26:56.015517 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 21666
I20260924 15:26:56.016577 21940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.016451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:36167 (local address 127.0.0.1:45716)
0924 15:26:56.016508 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.016549 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:36167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 15:26:56.016734 21453 catalog_manager.cc:4584] TS 22543368468f4e4990e724a891c8e171 (127.4.78.65:36167): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 10542ab54a8d4f4ca80634e48aaa8433: Network error: Client connection negotiation failed: client connection to 127.4.78.65:36167: connect: Connection refused (error 111)
I20260924 15:26:56.021248 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 21800
I20260924 15:26:56.025730 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 21437
2026-09-24T15:26:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2031 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T15:26:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:26:56Z Disabled control of system clock
I20260924 15:26:56.052959 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:37137
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:37137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:26:56.133175 21973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:56.133373 21973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:56.133394 21973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:56.134471 21973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:26:56.134506 21973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:56.134518 21973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:56.134531 21973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:26:56.134542 21973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:26:56.135778 21973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:37137
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:37137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:56.135955 21973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:56.136176 21973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:56.140820 21979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:56.140827 21981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:56.140827 21978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:56.141089 21973 server_base.cc:1034] running on GCE node
I20260924 15:26:56.141338 21973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:56.141569 21973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:56.142704 21973 hybrid_clock.cc:648] HybridClock initialized: now 1790263616142691 us; error 28 us; skew 500 ppm
I20260924 15:26:56.143942 21973 webserver.cc:460] Webserver started at http://127.4.78.126:35991/ using document root <none> and password file <none>
I20260924 15:26:56.144199 21973 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:56.144245 21973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:56.144394 21973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:56.145411 21973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "9c99ee3c2d1c435a8237d87c4c5355fc"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "4cb2fb3f4d5f4d00548ee126600f717f"
server_key_iv: "3c80b938fa38d63076ba4e42ce285190"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.145766 21973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "9c99ee3c2d1c435a8237d87c4c5355fc"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "4cb2fb3f4d5f4d00548ee126600f717f"
server_key_iv: "3c80b938fa38d63076ba4e42ce285190"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.147125 21973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:56.147830 21987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.148028 21973 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:56.148131 21973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "9c99ee3c2d1c435a8237d87c4c5355fc"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "4cb2fb3f4d5f4d00548ee126600f717f"
server_key_iv: "3c80b938fa38d63076ba4e42ce285190"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.148207 21973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:56.176644 21973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:56.176954 21973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:56.177138 21973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:56.181087 21973 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:37137
I20260924 15:26:56.181124 22040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:37137 every 8 connection(s)
I20260924 15:26:56.181463 21973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:26:56.182505 22041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:56.184693 22041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc: Bootstrap starting.
I20260924 15:26:56.185197 22041 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:56.185441 22041 log.cc:824] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:56.186094 22041 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc: No bootstrap required, opened a new log
I20260924 15:26:56.187376 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 21973
I20260924 15:26:56.187495 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:26:56.187666 4409 external_mini_cluster.cc:1428] Setting key 6698d1156775672a7ea4cb0c4a255b55
I20260924 15:26:56.187676 22041 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c99ee3c2d1c435a8237d87c4c5355fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37137 } }
I20260924 15:26:56.187872 22041 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:56.187925 22041 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c99ee3c2d1c435a8237d87c4c5355fc, State: Initialized, Role: FOLLOWER
I20260924 15:26:56.188072 22041 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c99ee3c2d1c435a8237d87c4c5355fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37137 } }
I20260924 15:26:56.188156 22041 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:26:56.188192 22041 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:26:56.188251 22041 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:56.188766 22041 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c99ee3c2d1c435a8237d87c4c5355fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37137 } }
I20260924 15:26:56.188849 22041 leader_election.cc:302] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c99ee3c2d1c435a8237d87c4c5355fc; no voters:
I20260924 15:26:56.188961 22041 leader_election.cc:288] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:26:56.189028 22046 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:56.189213 22046 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [term 1 LEADER]: Becoming Leader. State: Replica: 9c99ee3c2d1c435a8237d87c4c5355fc, State: Running, Role: LEADER
I20260924 15:26:56.189220 22041 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:26:56.189328 22046 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c99ee3c2d1c435a8237d87c4c5355fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37137 } }
I20260924 15:26:56.189685 22048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c99ee3c2d1c435a8237d87c4c5355fc. Latest consensus state: current_term: 1 leader_uuid: "9c99ee3c2d1c435a8237d87c4c5355fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c99ee3c2d1c435a8237d87c4c5355fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37137 } } }
I20260924 15:26:56.189783 22048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:56.189770 22047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c99ee3c2d1c435a8237d87c4c5355fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c99ee3c2d1c435a8237d87c4c5355fc" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 37137 } } }
I20260924 15:26:56.189895 22047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:56.190065 22055 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:26:56.190593 22055 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:26:56.192184 22055 catalog_manager.cc:1347] Generated new cluster ID: 76c2252834264329a43300c083127b64
I20260924 15:26:56.192236 22055 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:26:56.196063 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.188137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39300 (local address 127.4.78.126:37137)
0924 15:26:56.188371 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.188380 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.190774 (+ 2394us) server_negotiation.cc:407] Connection header received
0924 15:26:56.190939 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.190947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.191020 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.191172 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:56.191395 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.192679 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.193761 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.194375 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.194484 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.195252 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:56.195255 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:56.195255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:56.195459 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:56.195498 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:56.195511 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:56.195568 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.195665 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.195864 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260924 15:26:56.204180 22055 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:26:56.204802 22055 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:26:56.226416 22055 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc: Generated new TSK 0
I20260924 15:26:56.226637 22055 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:26:56.237929 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37137
--builtin_ntp_servers=127.4.78.84:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:26:56.319526 22065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:56.319736 22065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:56.319757 22065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:56.320927 22065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:56.320969 22065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:56.320993 22065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:26:56.322522 22065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:56.322779 22065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:56.323093 22065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:56.328076 22071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:56.328084 22073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:56.328091 22070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:56.328100 22065 server_base.cc:1034] running on GCE node
I20260924 15:26:56.328480 22065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:56.328760 22065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:56.329882 22065 hybrid_clock.cc:648] HybridClock initialized: now 1790263616329855 us; error 39 us; skew 500 ppm
I20260924 15:26:56.331049 22065 webserver.cc:460] Webserver started at http://127.4.78.65:35953/ using document root <none> and password file <none>
I20260924 15:26:56.331306 22065 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:56.331389 22065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:56.331549 22065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:56.332602 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "e2c7157790d04cb8b8db336c98baa031"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "9b1dc4a77cb3b71f322f8d2d9d872e38"
server_key_iv: "aeef23d25e374f550589b7c8ccd7c032"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.332966 22065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c7157790d04cb8b8db336c98baa031"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "9b1dc4a77cb3b71f322f8d2d9d872e38"
server_key_iv: "aeef23d25e374f550589b7c8ccd7c032"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.334417 22065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:26:56.335160 22079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.335337 22065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:26:56.335445 22065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "e2c7157790d04cb8b8db336c98baa031"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "9b1dc4a77cb3b71f322f8d2d9d872e38"
server_key_iv: "aeef23d25e374f550589b7c8ccd7c032"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.335527 22065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:56.351236 22065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:56.351562 22065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:56.351711 22065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:56.351960 22065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:56.352284 22065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:56.352339 22065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.352387 22065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:56.352414 22065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.358150 22065 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:46117
I20260924 15:26:56.358201 22192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:46117 every 8 connection(s)
I20260924 15:26:56.358506 22065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:26:56.362748 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 22065
I20260924 15:26:56.362855 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:26:56.363081 4409 external_mini_cluster.cc:1428] Setting key b137ee8d56999d351805a707b7ad0412
I20260924 15:26:56.363831 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.359162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:41739 (local address 127.4.78.126:37137)
0924 15:26:56.359243 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.359245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.359434 (+ 189us) server_negotiation.cc:407] Connection header received
0924 15:26:56.359913 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.359914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.359929 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.359977 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:56.360643 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.361139 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.362228 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.362412 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.362465 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.363543 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:56.363546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:56.363546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:56.363551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:56.363573 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:56.363577 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:56.363639 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.363722 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.363753 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:26:56.364039 22195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.359126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37137 (local address 127.4.78.65:41739)
0924 15:26:56.359329 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.359362 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:56.359830 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:56.360037 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:56.360040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:56.360048 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:56.360571 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.360576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.361277 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.361278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.362150 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.362153 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.362491 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.362492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.362650 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.363380 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:26:56.363380 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:56.363409 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:26:56.363431 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:56.363641 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:56.363647 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:56.363651 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:56.363724 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:26:56.363829 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":98,"threads_started":1}
I20260924 15:26:56.364114 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37137
--builtin_ntp_servers=127.4.78.84:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:26:56.364590 22193 heartbeater.cc:347] Connected to a master server at 127.4.78.126:37137
I20260924 15:26:56.364679 22193 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:56.364853 22193 heartbeater.cc:511] Master 127.4.78.126:37137 requested a full tablet report, sending...
I20260924 15:26:56.365388 22004 ts_manager.cc:194] Registered new tserver with Master: e2c7157790d04cb8b8db336c98baa031 (127.4.78.65:46117)
I20260924 15:26:56.366092 22004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.65:41739
W20260924 15:26:56.449774 22196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:56.450023 22196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:56.450045 22196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:56.451181 22196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:56.451265 22196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:56.451292 22196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:26:56.452579 22196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:56.452754 22196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:56.452979 22196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:56.457890 22202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:56.457903 22201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:56.457906 22204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:56.458037 22196 server_base.cc:1034] running on GCE node
I20260924 15:26:56.458240 22196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:56.458477 22196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:56.459623 22196 hybrid_clock.cc:648] HybridClock initialized: now 1790263616459603 us; error 40 us; skew 500 ppm
I20260924 15:26:56.460786 22196 webserver.cc:460] Webserver started at http://127.4.78.66:45315/ using document root <none> and password file <none>
I20260924 15:26:56.461035 22196 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:56.461102 22196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:56.461244 22196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:56.462354 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "19bb2ae9825b493b8c2b889d6d0918a1"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "44c29bed23a03c3895fe613d57f40ddd"
server_key_iv: "d57436552650e904875cb4a23cca58ad"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.462709 22196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "19bb2ae9825b493b8c2b889d6d0918a1"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "44c29bed23a03c3895fe613d57f40ddd"
server_key_iv: "d57436552650e904875cb4a23cca58ad"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.463979 22196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:26:56.464653 22210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.464814 22196 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:56.464920 22196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "19bb2ae9825b493b8c2b889d6d0918a1"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "44c29bed23a03c3895fe613d57f40ddd"
server_key_iv: "d57436552650e904875cb4a23cca58ad"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.465003 22196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:56.475443 22196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:56.475730 22196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:56.475883 22196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:56.476099 22196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:56.476431 22196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:56.476487 22196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.476528 22196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:56.476557 22196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.482414 22196 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:43929
I20260924 15:26:56.482465 22323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:43929 every 8 connection(s)
I20260924 15:26:56.482807 22196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:26:56.488096 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.483568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:50163 (local address 127.4.78.126:37137)
0924 15:26:56.483637 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.483639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.483860 (+ 221us) server_negotiation.cc:407] Connection header received
0924 15:26:56.484405 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.484407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.484420 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.484468 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:56.485157 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.485671 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.486709 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.486918 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.487007 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.487776 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:56.487777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:56.487778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:56.487783 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:56.487798 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:56.487802 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:56.487879 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.487983 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.488015 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:26:56.488343 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.483556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37137 (local address 127.4.78.66:50163)
0924 15:26:56.483734 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.483768 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:56.484288 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:56.484540 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:56.484549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:56.484557 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:56.485051 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.485056 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.485770 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.485772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.486615 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.486617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.487018 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.487019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.487152 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.487632 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:26:56.487632 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:56.487659 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:26:56.487682 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:56.487893 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:56.487899 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:56.487904 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:26:56.487986 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:26:56.488090 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260924 15:26:56.488868 22324 heartbeater.cc:347] Connected to a master server at 127.4.78.126:37137
I20260924 15:26:56.488971 22324 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:56.489111 22324 heartbeater.cc:511] Master 127.4.78.126:37137 requested a full tablet report, sending...
I20260924 15:26:56.489188 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 22196
I20260924 15:26:56.489313 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:26:56.489527 4409 external_mini_cluster.cc:1428] Setting key 6ee8b1c7098a1612bfd44b177dde27f7
I20260924 15:26:56.489530 22004 ts_manager.cc:194] Registered new tserver with Master: 19bb2ae9825b493b8c2b889d6d0918a1 (127.4.78.66:43929)
I20260924 15:26:56.490139 22004 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.66:50163
I20260924 15:26:56.490689 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37137
--builtin_ntp_servers=127.4.78.84:43069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:26:56.571062 22327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:56.571252 22327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:56.571273 22327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:56.572412 22327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:56.572451 22327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:56.572501 22327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:26:56.573901 22327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:37137
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:56.574136 22327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:56.574440 22327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:56.579305 22332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:56.579452 22333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:56.579301 22335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:56.579658 22327 server_base.cc:1034] running on GCE node
I20260924 15:26:56.579881 22327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:56.580111 22327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:56.581279 22327 hybrid_clock.cc:648] HybridClock initialized: now 1790263616581266 us; error 28 us; skew 500 ppm
I20260924 15:26:56.582669 22327 webserver.cc:460] Webserver started at http://127.4.78.67:33619/ using document root <none> and password file <none>
I20260924 15:26:56.582933 22327 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:56.583002 22327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:56.583139 22327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:56.584197 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "146d22453e5240389a12fe096c717f38"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "44822ce6b054a0146fe9d224de9d155b"
server_key_iv: "94d68b81b893284e1562be2bd296e756"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.584544 22327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "146d22453e5240389a12fe096c717f38"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "44822ce6b054a0146fe9d224de9d155b"
server_key_iv: "94d68b81b893284e1562be2bd296e756"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.585785 22327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:26:56.586565 22341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.586732 22327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:56.586810 22327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "146d22453e5240389a12fe096c717f38"
format_stamp: "Formatted at 2026-09-24 15:26:56 on dist-test-slave-l766"
server_key: "44822ce6b054a0146fe9d224de9d155b"
server_key_iv: "94d68b81b893284e1562be2bd296e756"
server_key_version: "encryptionkey@0"
I20260924 15:26:56.586869 22327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:56.595588 22327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:56.595813 22327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:56.595916 22327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:56.596072 22327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:56.596426 22327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:56.596457 22327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.596510 22327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:56.596542 22327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:56.602777 22327 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:42629
I20260924 15:26:56.602851 22455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:42629 every 8 connection(s)
I20260924 15:26:56.603176 22327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:26:56.605188 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 22327
I20260924 15:26:56.605283 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:26:56.605470 4409 external_mini_cluster.cc:1428] Setting key 6ea806cc9a7e8a3e45c3f80ef4b73f71
I20260924 15:26:56.608693 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.603943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:45841 (local address 127.4.78.126:37137)
0924 15:26:56.604001 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.604004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.604300 (+ 296us) server_negotiation.cc:407] Connection header received
0924 15:26:56.604778 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.604779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.604791 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.604855 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:56.605615 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.606349 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.607500 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.607681 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.607732 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.608484 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:56.608485 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:56.608485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:56.608489 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:56.608504 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:56.608508 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:56.608549 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.608612 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.608636 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:26:56.608908 22458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.603953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:37137 (local address 127.4.78.67:45841)
0924 15:26:56.604135 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.604194 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:56.604678 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:56.604910 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:56.604913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:56.604922 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:56.605523 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.605530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.606553 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.606554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.607425 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.607427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.607755 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.607756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.607869 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.608353 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:26:56.608354 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:56.608383 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:26:56.608406 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:56.608550 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:56.608552 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:56.608555 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:26:56.608605 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:26:56.608684 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260924 15:26:56.609385 22456 heartbeater.cc:347] Connected to a master server at 127.4.78.126:37137
I20260924 15:26:56.609464 22456 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:56.609606 22456 heartbeater.cc:511] Master 127.4.78.126:37137 requested a full tablet report, sending...
I20260924 15:26:56.609962 22003 ts_manager.cc:194] Registered new tserver with Master: 146d22453e5240389a12fe096c717f38 (127.4.78.67:42629)
I20260924 15:26:56.610348 22003 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.67:45841
I20260924 15:26:56.617380 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:26:56.621889 22045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.618893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39310 (local address 127.4.78.126:37137)
0924 15:26:56.618977 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.618979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.619091 (+ 112us) server_negotiation.cc:407] Connection header received
0924 15:26:56.619175 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.619175 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.619188 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.619243 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:56.619427 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.619905 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.620659 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.620870 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.620945 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.621550 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:56.621552 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:56.621552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:56.621557 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:56.621572 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:56.621576 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:56.621624 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.621720 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.621754 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:26:56.623374 22003 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:26:56.624972 22003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:26:56.631306 22465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.627150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:42629 (local address 127.0.0.1:50608)
0924 15:26:56.627591 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.627599 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:56.627642 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:56.627852 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:56.627853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:56.627855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:56.627994 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.627996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.629215 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.629216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.630118 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.630121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.631002 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.631003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.631079 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.631093 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:26:56.631172 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:26:56.631228 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":94,"threads_started":1}
I20260924 15:26:56.631426 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.627401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54112 (local address 127.4.78.65:46117)
0924 15:26:56.627576 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.627581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.627592 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:26:56.627633 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.627636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.627652 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.627751 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:56.627927 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.629066 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.630393 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.631087 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.631146 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.631227 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.631303 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.631339 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260924 15:26:56.631417 22464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.627146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:46117 (local address 127.0.0.1:54112)
0924 15:26:56.627324 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.627365 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:56.627430 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:56.627768 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:56.627772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:56.627774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:56.627841 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.627843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.629151 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.629152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.630307 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.630309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.631163 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.631164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.631225 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.631230 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:26:56.631277 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:26:56.631320 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":2,"thread_start_us":69,"threads_started":1}
I20260924 15:26:56.631737 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.627331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.4.78.67:42629)
0924 15:26:56.627537 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.627542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.627649 (+ 107us) server_negotiation.cc:407] Connection header received
0924 15:26:56.627695 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.627697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.627712 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.627784 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:56.628079 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.629134 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.630184 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.630908 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.630988 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.631080 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.631610 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.631650 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":102,"threads_started":1}
I20260924 15:26:56.632010 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.627156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:43929 (local address 127.0.0.1:56498)
0924 15:26:56.627700 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.627707 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:56.627769 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:56.628047 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:56.628048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:56.628050 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:56.628088 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.628089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.628985 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.628986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.630454 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.630455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.631332 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.631333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.631832 (+ 499us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.631838 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:26:56.631898 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:26:56.631938 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":29,"thread_start_us":102,"threads_started":1}
I20260924 15:26:56.632304 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.627393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56498 (local address 127.4.78.66:43929)
0924 15:26:56.627576 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.627581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.627837 (+ 256us) server_negotiation.cc:407] Connection header received
0924 15:26:56.627885 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.627888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.627906 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.627975 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:56.628184 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.628892 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.630537 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.631138 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.631209 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.631296 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.632179 (+ 883us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.632224 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":80,"threads_started":1}
I20260924 15:26:56.632225 22127 tablet_service.cc:1467] Processing CreateTablet for tablet e2abc01904e8448c99f33597920f7b11 (DEFAULT_TABLE table=test-table [id=7fdac68a2b594872950cdf3296238a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:56.632354 22389 tablet_service.cc:1467] Processing CreateTablet for tablet e2abc01904e8448c99f33597920f7b11 (DEFAULT_TABLE table=test-table [id=7fdac68a2b594872950cdf3296238a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:56.632607 22127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2abc01904e8448c99f33597920f7b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:56.632725 22389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2abc01904e8448c99f33597920f7b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:56.632903 22258 tablet_service.cc:1467] Processing CreateTablet for tablet e2abc01904e8448c99f33597920f7b11 (DEFAULT_TABLE table=test-table [id=7fdac68a2b594872950cdf3296238a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:56.633240 22258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2abc01904e8448c99f33597920f7b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:56.635766 22470 tablet_bootstrap.cc:492] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031: Bootstrap starting.
I20260924 15:26:56.636132 22471 tablet_bootstrap.cc:492] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: Bootstrap starting.
I20260924 15:26:56.636478 22470 tablet_bootstrap.cc:654] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:56.636468 22472 tablet_bootstrap.cc:492] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: Bootstrap starting.
I20260924 15:26:56.636848 22470 log.cc:824] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:56.637027 22471 tablet_bootstrap.cc:654] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:56.637143 22472 tablet_bootstrap.cc:654] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:56.637439 22471 log.cc:824] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:56.637465 22472 log.cc:824] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:56.637707 22470 tablet_bootstrap.cc:492] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031: No bootstrap required, opened a new log
I20260924 15:26:56.637799 22470 ts_tablet_manager.cc:1397] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:26:56.638195 22471 tablet_bootstrap.cc:492] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: No bootstrap required, opened a new log
I20260924 15:26:56.638190 22472 tablet_bootstrap.cc:492] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: No bootstrap required, opened a new log
I20260924 15:26:56.638257 22472 ts_tablet_manager.cc:1397] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:26:56.638257 22471 ts_tablet_manager.cc:1397] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:26:56.639896 22470 raft_consensus.cc:348] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.640058 22470 raft_consensus.cc:374] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:56.640102 22470 raft_consensus.cc:729] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2c7157790d04cb8b8db336c98baa031, State: Initialized, Role: FOLLOWER
I20260924 15:26:56.640255 22470 consensus_queue.cc:260] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.640543 22470 ts_tablet_manager.cc:1428] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:26:56.640679 22193 heartbeater.cc:503] Master 127.4.78.126:37137 was elected leader, sending a full tablet report...
I20260924 15:26:56.640910 22472 raft_consensus.cc:348] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.641089 22472 raft_consensus.cc:374] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:56.641141 22472 raft_consensus.cc:729] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19bb2ae9825b493b8c2b889d6d0918a1, State: Initialized, Role: FOLLOWER
I20260924 15:26:56.641268 22472 consensus_queue.cc:260] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.641299 22471 raft_consensus.cc:348] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.641458 22471 raft_consensus.cc:374] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:56.641500 22471 raft_consensus.cc:729] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 146d22453e5240389a12fe096c717f38, State: Initialized, Role: FOLLOWER
I20260924 15:26:56.641546 22472 ts_tablet_manager.cc:1428] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 15:26:56.641649 22324 heartbeater.cc:503] Master 127.4.78.126:37137 was elected leader, sending a full tablet report...
I20260924 15:26:56.641630 22471 consensus_queue.cc:260] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.641896 22471 ts_tablet_manager.cc:1428] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260924 15:26:56.642045 22456 heartbeater.cc:503] Master 127.4.78.126:37137 was elected leader, sending a full tablet report...
I20260924 15:26:56.647625 22477 raft_consensus.cc:482] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:56.647765 22477 raft_consensus.cc:504] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.648157 22477 leader_election.cc:288] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2c7157790d04cb8b8db336c98baa031 (127.4.78.65:46117), 146d22453e5240389a12fe096c717f38 (127.4.78.67:42629)
I20260924 15:26:56.651816 22469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.648436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:39247 (local address 127.4.78.65:46117)
0924 15:26:56.648477 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.648479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.648659 (+ 180us) server_negotiation.cc:407] Connection header received
0924 15:26:56.648690 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.648691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.648704 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.648758 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:56.648917 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.649746 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.651022 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.651437 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.651518 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.651529 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.651689 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.651729 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:26:56.651835 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.648357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:46117 (local address 127.4.78.66:39247)
0924 15:26:56.648577 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.648585 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:56.648637 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:56.648770 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:56.648771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:56.648772 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:56.648834 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.648836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.649846 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.649847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.650915 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.650918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.651518 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.651519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.651596 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.651600 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:56.651677 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:26:56.651718 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":67,"thread_start_us":94,"threads_started":1}
I20260924 15:26:56.651858 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.648512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:56853 (local address 127.4.78.67:42629)
0924 15:26:56.648573 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.648574 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.648583 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:26:56.648607 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.648608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.648615 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.648704 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:56.648896 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.649688 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.651062 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.651465 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.651562 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.651572 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.651735 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.651795 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:26:56.651858 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.648348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:42629 (local address 127.4.78.66:56853)
0924 15:26:56.648409 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.648423 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:56.648480 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:56.648715 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:56.648716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:56.648717 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:56.648799 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.648801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.649837 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.649839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.650969 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:56.650971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.651567 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.651568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:56.651639 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.651643 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:56.651725 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:26:56.651767 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":31}
I20260924 15:26:56.652242 22147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2abc01904e8448c99f33597920f7b11" candidate_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c7157790d04cb8b8db336c98baa031" is_pre_election: true
I20260924 15:26:56.652216 22410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2abc01904e8448c99f33597920f7b11" candidate_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146d22453e5240389a12fe096c717f38" is_pre_election: true
I20260924 15:26:56.652388 22410 raft_consensus.cc:2454] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bb2ae9825b493b8c2b889d6d0918a1 in term 0.
I20260924 15:26:56.652390 22147 raft_consensus.cc:2454] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19bb2ae9825b493b8c2b889d6d0918a1 in term 0.
I20260924 15:26:56.652638 22212 leader_election.cc:302] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19bb2ae9825b493b8c2b889d6d0918a1, e2c7157790d04cb8b8db336c98baa031; no voters:
I20260924 15:26:56.652807 22477 raft_consensus.cc:2781] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:26:56.652845 22477 raft_consensus.cc:482] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:26:56.652861 22477 raft_consensus.cc:3037] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:56.653556 22477 raft_consensus.cc:504] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.653743 22477 leader_election.cc:288] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [CANDIDATE]: Term 1 election: Requested vote from peers e2c7157790d04cb8b8db336c98baa031 (127.4.78.65:46117), 146d22453e5240389a12fe096c717f38 (127.4.78.67:42629)
I20260924 15:26:56.653910 22410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2abc01904e8448c99f33597920f7b11" candidate_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "146d22453e5240389a12fe096c717f38"
I20260924 15:26:56.653923 22147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2abc01904e8448c99f33597920f7b11" candidate_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2c7157790d04cb8b8db336c98baa031"
I20260924 15:26:56.653993 22410 raft_consensus.cc:3037] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:56.653993 22147 raft_consensus.cc:3037] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:56.654848 22147 raft_consensus.cc:2454] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bb2ae9825b493b8c2b889d6d0918a1 in term 1.
I20260924 15:26:56.654848 22410 raft_consensus.cc:2454] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19bb2ae9825b493b8c2b889d6d0918a1 in term 1.
I20260924 15:26:56.655026 22211 leader_election.cc:302] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 146d22453e5240389a12fe096c717f38, 19bb2ae9825b493b8c2b889d6d0918a1; no voters:
I20260924 15:26:56.655134 22477 raft_consensus.cc:2781] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:56.655328 22477 raft_consensus.cc:686] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 1 LEADER]: Becoming Leader. State: Replica: 19bb2ae9825b493b8c2b889d6d0918a1, State: Running, Role: LEADER
I20260924 15:26:56.655445 22477 consensus_queue.cc:237] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } }
I20260924 15:26:56.656338 22004 catalog_manager.cc:5495] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19bb2ae9825b493b8c2b889d6d0918a1 (127.4.78.66). New cstate: current_term: 1 leader_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19bb2ae9825b493b8c2b889d6d0918a1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 43929 } health_report { overall_health: HEALTHY } } }
I20260924 15:26:56.669754 22468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.667167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56510 (local address 127.4.78.66:43929)
0924 15:26:56.667229 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:56.667231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:56.667241 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:26:56.667289 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:56.667290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:56.667307 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:56.667396 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:26:56.667532 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.668075 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.668891 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:56.669092 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:56.669185 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:56.669383 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:26:56.669492 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:26:56.669594 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:56.669613 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 15:26:56.669633 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:26:56.671958 22147 raft_consensus.cc:1261] T e2abc01904e8448c99f33597920f7b11 P e2c7157790d04cb8b8db336c98baa031 [term 1 FOLLOWER]: Refusing update from remote peer 19bb2ae9825b493b8c2b889d6d0918a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:56.671937 22410 raft_consensus.cc:1261] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 1 FOLLOWER]: Refusing update from remote peer 19bb2ae9825b493b8c2b889d6d0918a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:56.672346 22480 consensus_queue.cc:1035] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2c7157790d04cb8b8db336c98baa031" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:56.672528 22477 consensus_queue.cc:1035] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "146d22453e5240389a12fe096c717f38" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:56.675751 22483 mvcc.cc:205] Tried to move back new op lower bound from 7332919773884534784 to 7332919773821403136. Current Snapshot: MvccSnapshot[applied={T|T < 7332919773884534784}]
I20260924 15:26:56.676034 22484 mvcc.cc:205] Tried to move back new op lower bound from 7332919773884534784 to 7332919773821403136. Current Snapshot: MvccSnapshot[applied={T|T < 7332919773884534784}]
I20260924 15:26:56.676299 22485 mvcc.cc:205] Tried to move back new op lower bound from 7332919773884534784 to 7332919773821403136. Current Snapshot: MvccSnapshot[applied={T|T < 7332919773884534784}]
I20260924 15:26:56.678412 22003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:56.678516 22003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:56.679261 22003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c99ee3c2d1c435a8237d87c4c5355fc: Sending DeleteTablet for 3 replicas of tablet e2abc01904e8448c99f33597920f7b11
I20260924 15:26:56.679597 22258 tablet_service.cc:1514] Processing DeleteTablet for tablet e2abc01904e8448c99f33597920f7b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:56 UTC) from {username='slave'} at 127.0.0.1:56498
I20260924 15:26:56.679605 22389 tablet_service.cc:1514] Processing DeleteTablet for tablet e2abc01904e8448c99f33597920f7b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:56 UTC) from {username='slave'} at 127.0.0.1:50608
I20260924 15:26:56.679634 22127 tablet_service.cc:1514] Processing DeleteTablet for tablet e2abc01904e8448c99f33597920f7b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:56 UTC) from {username='slave'} at 127.0.0.1:54112
I20260924 15:26:56.679872 22493 tablet_replica.cc:330] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: stopping tablet replica
I20260924 15:26:56.680064 22495 tablet_replica.cc:330] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: stopping tablet replica
I20260924 15:26:56.680188 22495 raft_consensus.cc:2229] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:26:56.680230 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 22065
I20260924 15:26:56.680326 22495 raft_consensus.cc:2258] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:56.680500 22493 raft_consensus.cc:2229] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:56.681015 22493 raft_consensus.cc:2258] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:56.681659 22493 ts_tablet_manager.cc:1905] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:56.681659 22495 ts_tablet_manager.cc:1905] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:56.683478 22493 ts_tablet_manager.cc:1918] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:56.683478 22495 ts_tablet_manager.cc:1918] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:56.683557 22493 log.cc:1196] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/e2abc01904e8448c99f33597920f7b11
I20260924 15:26:56.683768 22495 log.cc:1196] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/e2abc01904e8448c99f33597920f7b11
I20260924 15:26:56.683825 22493 ts_tablet_manager.cc:1939] T e2abc01904e8448c99f33597920f7b11 P 146d22453e5240389a12fe096c717f38: Deleting consensus metadata
I20260924 15:26:56.684149 22495 ts_tablet_manager.cc:1939] T e2abc01904e8448c99f33597920f7b11 P 19bb2ae9825b493b8c2b889d6d0918a1: Deleting consensus metadata
I20260924 15:26:56.684398 21988 catalog_manager.cc:4841] TS 146d22453e5240389a12fe096c717f38 (127.4.78.67:42629): tablet e2abc01904e8448c99f33597920f7b11 (table test-table [id=7fdac68a2b594872950cdf3296238a55]) successfully deleted
I20260924 15:26:56.684561 21991 catalog_manager.cc:4841] TS 19bb2ae9825b493b8c2b889d6d0918a1 (127.4.78.66:43929): tablet e2abc01904e8448c99f33597920f7b11 (table test-table [id=7fdac68a2b594872950cdf3296238a55]) successfully deleted
W20260924 15:26:56.688380 21989 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:46117 (error 108)
I20260924 15:26:56.688611 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 22196
I20260924 15:26:56.689265 22466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:56.688750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:46117 (local address 127.0.0.1:54118)
0924 15:26:56.689180 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:56.689226 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:46117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394}
W20260924 15:26:56.689469 21989 catalog_manager.cc:4584] TS e2c7157790d04cb8b8db336c98baa031 (127.4.78.65:46117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2abc01904e8448c99f33597920f7b11: Network error: Client connection negotiation failed: client connection to 127.4.78.65:46117: connect: Connection refused (error 111)
I20260924 15:26:56.695124 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 22327
I20260924 15:26:56.699602 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 21973
2026-09-24T15:26:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (673 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:26:56 dist-test-slave-l766 krb5kdc[22499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:26:56 dist-test-slave-l766 krb5kdc[22499](info): set up 2 sockets
Sep 24 15:26:56 dist-test-slave-l766 krb5kdc[22499](info): commencing operation
krb5kdc: starting...
W20260924 15:26:57.736490 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:26:57 dist-test-slave-l766 krb5kdc[22499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:26:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:26:57.857717 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35141
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:57.938418 22512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:57.938612 22512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:57.938634 22512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:57.939705 22512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:26:57.939736 22512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:57.939750 22512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:57.939764 22512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:26:57.939774 22512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:26:57.941128 22512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:35141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:57.941313 22512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:57.941474 22512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:57.946192 22518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:57.946158 22520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:57.946158 22517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:57.946360 22512 server_base.cc:1034] running on GCE node
I20260924 15:26:57.946583 22512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:57.946861 22512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:57.948007 22512 hybrid_clock.cc:648] HybridClock initialized: now 1790263617947994 us; error 28 us; skew 500 ppm
Sep 24 15:26:57 dist-test-slave-l766 krb5kdc[22499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263617, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:57.950526 22512 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:26:57.950999 22512 webserver.cc:460] Webserver started at http://127.4.78.126:44695/ using document root <none> and password file <none>
I20260924 15:26:57.951221 22512 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:57.951290 22512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:57.951447 22512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:57.952541 22512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "7ada860ca3ae43329a61a419c43a1dfb"
format_stamp: "Formatted at 2026-09-24 15:26:57 on dist-test-slave-l766"
server_key: "d69c7936ca376d3f3129400c03f09d6e"
server_key_iv: "fe8c9d315cd8697bc939486829b2fd7e"
server_key_version: "encryptionkey@0"
I20260924 15:26:57.952905 22512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "7ada860ca3ae43329a61a419c43a1dfb"
format_stamp: "Formatted at 2026-09-24 15:26:57 on dist-test-slave-l766"
server_key: "d69c7936ca376d3f3129400c03f09d6e"
server_key_iv: "fe8c9d315cd8697bc939486829b2fd7e"
server_key_version: "encryptionkey@0"
I20260924 15:26:57.954357 22512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:26:57.955092 22527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:57.955261 22512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:26:57.955382 22512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "7ada860ca3ae43329a61a419c43a1dfb"
format_stamp: "Formatted at 2026-09-24 15:26:57 on dist-test-slave-l766"
server_key: "d69c7936ca376d3f3129400c03f09d6e"
server_key_iv: "fe8c9d315cd8697bc939486829b2fd7e"
server_key_version: "encryptionkey@0"
I20260924 15:26:57.955457 22512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:57.987432 22512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:57.989406 22512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:57.989580 22512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:57.993700 22512 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:35141
I20260924 15:26:57.993745 22579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:35141 every 8 connection(s)
I20260924 15:26:57.994122 22512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:26:57.994962 22580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:57.997290 22580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb: Bootstrap starting.
I20260924 15:26:57.997901 22580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:57.998202 22580 log.cc:824] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:57.998916 22580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb: No bootstrap required, opened a new log
I20260924 15:26:58.000525 22580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ada860ca3ae43329a61a419c43a1dfb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35141 } }
I20260924 15:26:58.000662 22580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:58.000685 22580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ada860ca3ae43329a61a419c43a1dfb, State: Initialized, Role: FOLLOWER
I20260924 15:26:58.000869 22580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ada860ca3ae43329a61a419c43a1dfb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35141 } }
I20260924 15:26:58.000931 22580 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:26:58.000989 22580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:26:58.001042 22580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:58.001590 22580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ada860ca3ae43329a61a419c43a1dfb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35141 } }
I20260924 15:26:58.001677 22580 leader_election.cc:302] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ada860ca3ae43329a61a419c43a1dfb; no voters:
I20260924 15:26:58.001892 22580 leader_election.cc:288] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:26:58.001924 22583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:58.002099 22583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [term 1 LEADER]: Becoming Leader. State: Replica: 7ada860ca3ae43329a61a419c43a1dfb, State: Running, Role: LEADER
I20260924 15:26:58.002144 22580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:26:58.002190 22583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ada860ca3ae43329a61a419c43a1dfb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35141 } }
I20260924 15:26:58.002496 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 22512
I20260924 15:26:58.002640 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:26:58.002879 4409 external_mini_cluster.cc:1428] Setting key fcb6531ce01d47151b036a2629dab744
I20260924 15:26:58.003257 22584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ada860ca3ae43329a61a419c43a1dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ada860ca3ae43329a61a419c43a1dfb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35141 } } }
I20260924 15:26:58.003353 22584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:58.003748 22585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ada860ca3ae43329a61a419c43a1dfb. Latest consensus state: current_term: 1 leader_uuid: "7ada860ca3ae43329a61a419c43a1dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ada860ca3ae43329a61a419c43a1dfb" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 35141 } } }
I20260924 15:26:58.003865 22585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb [sys.catalog]: This master's current role is: LEADER
W20260924 15:26:58.004163 22600 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:26:58.004222 22600 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:26:58.004281 22598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:26:58.004459 22598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:26:58.006098 22598 catalog_manager.cc:1347] Generated new cluster ID: 68d818c99f254102945509b995f46136
I20260924 15:26:58.006146 22598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:26:58 dist-test-slave-l766 krb5kdc[22499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263617, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:58.013975 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.003518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.4.78.126:35141)
0924 15:26:58.003958 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.003968 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.004001 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:26:58.004123 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.004129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.004186 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.005170 (+ 984us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:58.005252 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.006443 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.007275 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.007708 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.007842 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.010213 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:58.010218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:58.010219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:58.010231 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:58.012366 (+ 2135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.012745 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.012748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.012749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.012851 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.013065 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.013066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.013067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.013308 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:58.013485 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.013580 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.013737 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":66,"threads_started":1}
I20260924 15:26:58.015607 22598 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:26:58.016114 22598 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:26:58.020381 22598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb: Generated new TSK 0
I20260924 15:26:58.020561 22598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:26:58.050702 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35141
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:58.131085 22606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:58.131287 22606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:58.131310 22606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:58.132371 22606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:58.132403 22606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:58.132428 22606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:26:58.133752 22606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:58.133944 22606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:58.134148 22606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:58.139065 22611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:58.139194 22606 server_base.cc:1034] running on GCE node
W20260924 15:26:58.139073 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.139281 22612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:58.139499 22606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:58.139788 22606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:58.140933 22606 hybrid_clock.cc:648] HybridClock initialized: now 1790263618140910 us; error 35 us; skew 500 ppm
Sep 24 15:26:58 dist-test-slave-l766 krb5kdc[22499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263618, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:58.143440 22606 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:26:58.143853 22606 webserver.cc:460] Webserver started at http://127.4.78.65:36415/ using document root <none> and password file <none>
I20260924 15:26:58.144062 22606 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:58.144106 22606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:58.144245 22606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:58.145249 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "40b09485fa704e9d81a2c7e46034bd82"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "386bce6b3196329a98bc2bf0c307d266"
server_key_iv: "eed9207413a2c077b46f4eed01832bc4"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.145591 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "40b09485fa704e9d81a2c7e46034bd82"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "386bce6b3196329a98bc2bf0c307d266"
server_key_iv: "eed9207413a2c077b46f4eed01832bc4"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.146955 22606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:26:58.147677 22621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.147900 22606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:58.147990 22606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "40b09485fa704e9d81a2c7e46034bd82"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "386bce6b3196329a98bc2bf0c307d266"
server_key_iv: "eed9207413a2c077b46f4eed01832bc4"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.148042 22606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:58.164976 22606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:58.166884 22606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:58.167019 22606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:58.167269 22606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:58.167577 22606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:58.167609 22606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.167630 22606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:58.167663 22606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.173065 22606 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:41277
I20260924 15:26:58.173146 22734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:41277 every 8 connection(s)
I20260924 15:26:58.173444 22606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:26:58.175345 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 22606
I20260924 15:26:58.175464 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:26:58.175652 4409 external_mini_cluster.cc:1428] Setting key 1241e4411bbc18b0b29601dae92df84c
Sep 24 15:26:58 dist-test-slave-l766 krb5kdc[22499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263618, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:58.183079 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.174230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:52977 (local address 127.4.78.126:35141)
0924 15:26:58.174352 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.174355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.174511 (+ 156us) server_negotiation.cc:407] Connection header received
0924 15:26:58.175027 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.175028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.175044 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.175096 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:58.176122 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.176626 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.177712 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.177985 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.178043 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.180117 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:58.180119 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:58.180120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:58.180127 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:58.181618 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.182118 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.182119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.182119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.182171 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.182457 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.182458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.182458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.182564 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:58.182619 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.182949 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.183015 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 15:26:58.183346 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.174195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35141 (local address 127.4.78.65:52977)
0924 15:26:58.174381 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.174424 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.174923 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.175163 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.175167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.175836 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:58.176019 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.176024 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.176728 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.176729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.177623 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.177626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.178067 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.178068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.178228 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.178690 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:58.178698 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:58.180023 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:58.181702 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:58.181711 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:58.181713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:58.182045 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:58.182243 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:58.182244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:58.182244 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:58.182386 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:58.182634 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:58.182640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:58.182808 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.182930 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.183091 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260924 15:26:58.183888 22735 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35141
I20260924 15:26:58.183976 22735 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:58.184156 22735 heartbeater.cc:511] Master 127.4.78.126:35141 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
I20260924 15:26:58.184763 22544 ts_manager.cc:194] Registered new tserver with Master: 40b09485fa704e9d81a2c7e46034bd82 (127.4.78.65:41277)
I20260924 15:26:58.185391 22544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:52977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:26:58.201385 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35141
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:58.280862 22740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:58.281054 22740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:58.281077 22740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:58.282295 22740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:58.282335 22740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:58.282362 22740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:26:58.283778 22740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:58.284011 22740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:58.284252 22740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:58.289248 22746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:58.289316 22740 server_base.cc:1034] running on GCE node
W20260924 15:26:58.289124 22745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.289237 22748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:58.289680 22740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:58.289945 22740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:58.291086 22740 hybrid_clock.cc:648] HybridClock initialized: now 1790263618291050 us; error 53 us; skew 500 ppm
Sep 24 15:26:58 dist-test-slave-l766 krb5kdc[22499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263618, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:58.293481 22740 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:26:58.293992 22740 webserver.cc:460] Webserver started at http://127.4.78.66:33437/ using document root <none> and password file <none>
I20260924 15:26:58.294237 22740 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:58.294310 22740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:58.294453 22740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:58.295454 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "49155d1bddf54d48998eefbf164bffe8"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "4e2f5908920f48cbbb4298bfcf66d645"
server_key_iv: "7a06044666419e58286201a62789cfc7"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.295805 22740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "49155d1bddf54d48998eefbf164bffe8"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "4e2f5908920f48cbbb4298bfcf66d645"
server_key_iv: "7a06044666419e58286201a62789cfc7"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.297083 22740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:26:58.297802 22755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.298048 22740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:26:58.298144 22740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "49155d1bddf54d48998eefbf164bffe8"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "4e2f5908920f48cbbb4298bfcf66d645"
server_key_iv: "7a06044666419e58286201a62789cfc7"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.298229 22740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:58.309614 22740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:58.311722 22740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:58.311892 22740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:58.312124 22740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:58.312446 22740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:58.312501 22740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.312543 22740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:58.312572 22740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.318562 22740 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:33817
I20260924 15:26:58.318614 22868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:33817 every 8 connection(s)
I20260924 15:26:58.318920 22740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:26:58 dist-test-slave-l766 krb5kdc[22499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263618, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:58.325938 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 22740
I20260924 15:26:58.326063 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:26:58.326299 4409 external_mini_cluster.cc:1428] Setting key 64057322b82562e19168b295e54cfc6f
I20260924 15:26:58.328274 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.319631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:45169 (local address 127.4.78.126:35141)
0924 15:26:58.319676 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.319678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.319908 (+ 230us) server_negotiation.cc:407] Connection header received
0924 15:26:58.320366 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.320367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.320378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.320440 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:58.321398 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.321942 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.322901 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.323078 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.323128 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.325248 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:58.325250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:58.325250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:58.325257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:58.326917 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.327296 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.327298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.327298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.327355 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.327638 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.327639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.327640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.327751 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:58.327830 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.328102 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.328184 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:26:58.328553 22871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.319585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35141 (local address 127.4.78.66:45169)
0924 15:26:58.319747 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.319790 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.320279 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.320523 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.320526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.321086 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:58.321278 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.321283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.322030 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.322031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.322827 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.322829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.323146 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.323147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.323240 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.323782 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:58.323790 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:58.325144 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:58.326982 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:58.326989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:58.326992 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:58.327210 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:58.327427 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:58.327428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:58.327428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:58.327556 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:58.327826 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:58.327833 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:58.327973 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.328081 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.328238 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260924 15:26:58.329092 22869 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35141
I20260924 15:26:58.329216 22869 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:58.329407 22869 heartbeater.cc:511] Master 127.4.78.126:35141 requested a full tablet report, sending...
I20260924 15:26:58.329850 22544 ts_manager.cc:194] Registered new tserver with Master: 49155d1bddf54d48998eefbf164bffe8 (127.4.78.66:33817)
I20260924 15:26:58.330240 22544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:45169
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:26:58.352293 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35141
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:26:58.433770 22874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:58.434049 22874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:58.434072 22874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:58.435155 22874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:58.435187 22874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:58.435213 22874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:26:58.436617 22874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:58.436789 22874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:58.436954 22874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:58.441852 22879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.441936 22882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.441852 22880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:58.442021 22874 server_base.cc:1034] running on GCE node
I20260924 15:26:58.442276 22874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:58.442552 22874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:58.443719 22874 hybrid_clock.cc:648] HybridClock initialized: now 1790263618443696 us; error 34 us; skew 500 ppm
Sep 24 15:26:58 dist-test-slave-l766 krb5kdc[22499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263618, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:26:58.446167 22874 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:26:58.446625 22874 webserver.cc:460] Webserver started at http://127.4.78.67:42681/ using document root <none> and password file <none>
I20260924 15:26:58.446857 22874 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:58.446903 22874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:58.447042 22874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:58.448071 22874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "e01141f462a34bf1a6d4383179152664"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "3464a409b69a09c603dc8bb3679450d3"
server_key_iv: "0edb66a67e4a5b3a9ddc7db51fe2a28c"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.448426 22874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "e01141f462a34bf1a6d4383179152664"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "3464a409b69a09c603dc8bb3679450d3"
server_key_iv: "0edb66a67e4a5b3a9ddc7db51fe2a28c"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.449707 22874 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:26:58.450459 22889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.450644 22874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:26:58.450714 22874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "e01141f462a34bf1a6d4383179152664"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "3464a409b69a09c603dc8bb3679450d3"
server_key_iv: "0edb66a67e4a5b3a9ddc7db51fe2a28c"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.450803 22874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:58.475996 22874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:58.478408 22874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:58.478524 22874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:58.478719 22874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:58.479048 22874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:58.479104 22874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.479139 22874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:58.479187 22874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.485070 22874 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:41227
I20260924 15:26:58.485112 23002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:41227 every 8 connection(s)
I20260924 15:26:58.485466 22874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:26:58.487087 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 22874
I20260924 15:26:58.487202 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:26:58.487377 4409 external_mini_cluster.cc:1428] Setting key 1e4e8e239cb023ec29f6a1994dbe7af9
Sep 24 15:26:58 dist-test-slave-l766 krb5kdc[22499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263618, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:26:58.494679 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.486225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:59131 (local address 127.4.78.126:35141)
0924 15:26:58.486290 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.486292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.486540 (+ 248us) server_negotiation.cc:407] Connection header received
0924 15:26:58.487063 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.487064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.487076 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.487128 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:58.488259 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.488747 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.489752 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.490010 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.490094 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.492131 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:58.492132 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:58.492133 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:58.492140 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:58.493281 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.493673 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.493674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.493675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.493734 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.494028 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.494029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.494030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.494143 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:58.494235 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.494509 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.494589 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:26:58.494868 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.486206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:35141 (local address 127.4.78.67:59131)
0924 15:26:58.486384 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.486447 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.486958 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.487214 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.487218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.487918 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:58.488166 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.488174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.488848 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.488849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.489659 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.489661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.490105 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.490106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.490230 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.490733 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:26:58.490741 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:58.492037 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:58.493383 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:58.493389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:58.493392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:58.493598 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:58.493801 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:26:58.493802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:26:58.493803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:26:58.493960 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:26:58.494226 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:58.494232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:58.494376 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.494485 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.494638 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260924 15:26:58.495337 23003 heartbeater.cc:347] Connected to a master server at 127.4.78.126:35141
I20260924 15:26:58.495436 23003 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:58.495615 23003 heartbeater.cc:511] Master 127.4.78.126:35141 requested a full tablet report, sending...
I20260924 15:26:58.496022 22544 ts_manager.cc:194] Registered new tserver with Master: e01141f462a34bf1a6d4383179152664 (127.4.78.67:41227)
I20260924 15:26:58.496402 22544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:59131
I20260924 15:26:58.499111 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:26:58.506636 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.500484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.4.78.126:35141)
0924 15:26:58.500550 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.500553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.500655 (+ 102us) server_negotiation.cc:407] Connection header received
0924 15:26:58.500693 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.500694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.500702 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.500745 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:58.501356 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.501838 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.502655 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.502864 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.502929 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.504396 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:58.504398 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:58.504398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:26:58.504403 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:58.505530 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.505935 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.505936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.505936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.505984 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:26:58.506175 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:26:58.506175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:26:58.506176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:26:58.506291 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:58.506373 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.506482 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.506529 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:26:58.508260 22544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:26:58.509795 22544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:26:58.516125 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.511975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:41227 (local address 127.0.0.1:55536)
0924 15:26:58.512283 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.512331 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.512390 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.512767 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.512771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.512773 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:58.512862 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.512863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.513752 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.513753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.515073 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.515076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.515810 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.515811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.515891 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.515905 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.516001 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.516054 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":55,"threads_started":1}
I20260924 15:26:58.516139 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.512137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55536 (local address 127.4.78.67:41227)
0924 15:26:58.512353 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.512359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.512376 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:26:58.512532 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.512535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.512552 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.512628 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:58.512950 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.513651 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.515158 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.515721 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.515808 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.515877 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.516017 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.516061 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":108,"threads_started":1}
I20260924 15:26:58.516538 23013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.512169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:33817 (local address 127.0.0.1:39008)
0924 15:26:58.512476 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.512485 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.512538 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.512769 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.512776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.512778 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:58.512863 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.512866 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.514206 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.514207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.515263 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.515266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.516316 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.516317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.516396 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.516400 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.516440 (+ 40us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.516469 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":57,"thread_start_us":51,"threads_started":1}
I20260924 15:26:58.516568 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.512142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45068 (local address 127.4.78.65:41277)
0924 15:26:58.512661 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.512666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.512675 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:26:58.512717 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.512719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.512735 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.512811 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:58.513011 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.514113 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.515390 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.516091 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.516163 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.516224 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.516433 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.516483 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":81,"threads_started":1}
I20260924 15:26:58.516539 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.512026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:41277 (local address 127.0.0.1:45068)
0924 15:26:58.512421 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.512431 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.512486 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.512896 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.512897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.512900 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:58.512944 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.512945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.514217 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.514218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.515282 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.515285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.516266 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.516268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.516338 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.516342 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.516401 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.516451 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":20,"thread_start_us":51,"threads_started":1}
I20260924 15:26:58.517068 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.512299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.4.78.66:33817)
0924 15:26:58.512496 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.512500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.512530 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:26:58.512601 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.512603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.512618 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.512686 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:58.512946 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.514080 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.515363 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.516092 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.516162 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.516936 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.516969 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.517007 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":98,"threads_started":1}
I20260924 15:26:58.517077 22937 tablet_service.cc:1467] Processing CreateTablet for tablet 081c77157c454503949d13915f29e97a (DEFAULT_TABLE table=test-table [id=1968d17a8cf144d4a09b4da4e4922676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:58.517445 22937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081c77157c454503949d13915f29e97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:58.517382 22669 tablet_service.cc:1467] Processing CreateTablet for tablet 081c77157c454503949d13915f29e97a (DEFAULT_TABLE table=test-table [id=1968d17a8cf144d4a09b4da4e4922676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:58.517676 22669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081c77157c454503949d13915f29e97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:58.517684 22803 tablet_service.cc:1467] Processing CreateTablet for tablet 081c77157c454503949d13915f29e97a (DEFAULT_TABLE table=test-table [id=1968d17a8cf144d4a09b4da4e4922676]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:58.518064 22803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 081c77157c454503949d13915f29e97a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:58.520515 23017 tablet_bootstrap.cc:492] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: Bootstrap starting.
I20260924 15:26:58.520866 23018 tablet_bootstrap.cc:492] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82: Bootstrap starting.
I20260924 15:26:58.521410 23017 tablet_bootstrap.cc:654] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:58.521476 23018 tablet_bootstrap.cc:654] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:58.521976 23019 tablet_bootstrap.cc:492] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: Bootstrap starting.
I20260924 15:26:58.522085 23017 log.cc:824] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:58.522085 23018 log.cc:824] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:58.522644 23019 tablet_bootstrap.cc:654] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:58.522904 23017 tablet_bootstrap.cc:492] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: No bootstrap required, opened a new log
I20260924 15:26:58.522979 23017 ts_tablet_manager.cc:1397] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 15:26:58.523096 23018 tablet_bootstrap.cc:492] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82: No bootstrap required, opened a new log
I20260924 15:26:58.523159 23018 ts_tablet_manager.cc:1397] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:26:58.523228 23019 log.cc:824] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:58.524025 23019 tablet_bootstrap.cc:492] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: No bootstrap required, opened a new log
I20260924 15:26:58.524106 23019 ts_tablet_manager.cc:1397] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:26:58.524963 23017 raft_consensus.cc:348] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.525018 23018 raft_consensus.cc:348] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.525154 23018 raft_consensus.cc:374] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:58.525154 23017 raft_consensus.cc:374] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:58.525192 23018 raft_consensus.cc:729] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40b09485fa704e9d81a2c7e46034bd82, State: Initialized, Role: FOLLOWER
I20260924 15:26:58.525192 23017 raft_consensus.cc:729] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49155d1bddf54d48998eefbf164bffe8, State: Initialized, Role: FOLLOWER
I20260924 15:26:58.525314 23017 consensus_queue.cc:260] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.525369 23018 consensus_queue.cc:260] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.525624 23017 ts_tablet_manager.cc:1428] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:26:58.525624 23018 ts_tablet_manager.cc:1428] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:26:58.525548 23019 raft_consensus.cc:348] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.525717 23019 raft_consensus.cc:374] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:58.525743 22735 heartbeater.cc:503] Master 127.4.78.126:35141 was elected leader, sending a full tablet report...
I20260924 15:26:58.525753 22869 heartbeater.cc:503] Master 127.4.78.126:35141 was elected leader, sending a full tablet report...
I20260924 15:26:58.525755 23019 raft_consensus.cc:729] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e01141f462a34bf1a6d4383179152664, State: Initialized, Role: FOLLOWER
I20260924 15:26:58.525888 23019 consensus_queue.cc:260] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.526149 23019 ts_tablet_manager.cc:1428] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:26:58.526293 23003 heartbeater.cc:503] Master 127.4.78.126:35141 was elected leader, sending a full tablet report...
I20260924 15:26:58.532276 23025 raft_consensus.cc:482] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:58.532394 23025 raft_consensus.cc:504] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.532723 23025 leader_election.cc:288] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49155d1bddf54d48998eefbf164bffe8 (127.4.78.66:33817), 40b09485fa704e9d81a2c7e46034bd82 (127.4.78.65:41277)
I20260924 15:26:58.535431 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.532901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:38679 (local address 127.4.78.66:33817)
0924 15:26:58.532945 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.532947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.532958 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:26:58.532984 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.532985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.532995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.533044 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:58.533229 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.533749 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.534683 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.535087 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.535140 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.535148 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.535321 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.535364 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:26:58.535431 23005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.532843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:33817 (local address 127.4.78.67:38679)
0924 15:26:58.532890 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.532900 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.532937 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.533080 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.533081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.533082 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:58.533146 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.533147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.533860 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.533861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.534596 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.534598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.535161 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.535163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.535248 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.535251 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.535307 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.535344 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":7}
I20260924 15:26:58.535526 23014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.533009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:45127 (local address 127.4.78.65:41277)
0924 15:26:58.533052 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.533054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.533130 (+ 76us) server_negotiation.cc:407] Connection header received
0924 15:26:58.533176 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.533177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.533188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.533237 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:58.533360 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.533778 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.534773 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.535156 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.535223 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.535233 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.535414 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.535456 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:26:58.535801 22823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081c77157c454503949d13915f29e97a" candidate_uuid: "e01141f462a34bf1a6d4383179152664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49155d1bddf54d48998eefbf164bffe8" is_pre_election: true
I20260924 15:26:58.535854 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.532900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:41277 (local address 127.4.78.67:45127)
0924 15:26:58.533077 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.533086 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.533136 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.533241 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.533241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.533243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:58.533307 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.533308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.533906 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.533906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.534687 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.534689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.535240 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.535240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.535335 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.535339 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.535397 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.535432 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
I20260924 15:26:58.535913 22823 raft_consensus.cc:2454] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01141f462a34bf1a6d4383179152664 in term 0.
I20260924 15:26:58.536183 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081c77157c454503949d13915f29e97a" candidate_uuid: "e01141f462a34bf1a6d4383179152664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b09485fa704e9d81a2c7e46034bd82" is_pre_election: true
I20260924 15:26:58.536254 22891 leader_election.cc:302] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49155d1bddf54d48998eefbf164bffe8, e01141f462a34bf1a6d4383179152664; no voters:
I20260924 15:26:58.536310 22689 raft_consensus.cc:2454] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e01141f462a34bf1a6d4383179152664 in term 0.
I20260924 15:26:58.536396 23025 raft_consensus.cc:2781] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:26:58.536500 23025 raft_consensus.cc:482] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:26:58.536530 23025 raft_consensus.cc:3037] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:58.537274 23025 raft_consensus.cc:504] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.537417 23025 leader_election.cc:288] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [CANDIDATE]: Term 1 election: Requested vote from peers 49155d1bddf54d48998eefbf164bffe8 (127.4.78.66:33817), 40b09485fa704e9d81a2c7e46034bd82 (127.4.78.65:41277)
I20260924 15:26:58.537550 22689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081c77157c454503949d13915f29e97a" candidate_uuid: "e01141f462a34bf1a6d4383179152664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40b09485fa704e9d81a2c7e46034bd82"
I20260924 15:26:58.537617 22689 raft_consensus.cc:3037] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:58.537586 22823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "081c77157c454503949d13915f29e97a" candidate_uuid: "e01141f462a34bf1a6d4383179152664" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49155d1bddf54d48998eefbf164bffe8"
I20260924 15:26:58.537679 22823 raft_consensus.cc:3037] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:58.538381 22689 raft_consensus.cc:2454] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01141f462a34bf1a6d4383179152664 in term 1.
I20260924 15:26:58.538439 22823 raft_consensus.cc:2454] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e01141f462a34bf1a6d4383179152664 in term 1.
I20260924 15:26:58.538592 22892 leader_election.cc:302] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40b09485fa704e9d81a2c7e46034bd82, e01141f462a34bf1a6d4383179152664; no voters:
I20260924 15:26:58.538671 23025 raft_consensus.cc:2781] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:58.538810 23025 raft_consensus.cc:686] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 1 LEADER]: Becoming Leader. State: Replica: e01141f462a34bf1a6d4383179152664, State: Running, Role: LEADER
I20260924 15:26:58.538909 23025 consensus_queue.cc:237] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } }
I20260924 15:26:58.539654 22543 catalog_manager.cc:5495] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 reported cstate change: term changed from 0 to 1, leader changed from <none> to e01141f462a34bf1a6d4383179152664 (127.4.78.67). New cstate: current_term: 1 leader_uuid: "e01141f462a34bf1a6d4383179152664" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e01141f462a34bf1a6d4383179152664" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 } health_report { overall_health: UNKNOWN } } }
I20260924 15:26:58.547112 23015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.544984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55544 (local address 127.4.78.67:41227)
0924 15:26:58.545033 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.545035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.545044 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:26:58.545071 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.545072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.545094 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.545151 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:26:58.545290 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.545720 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.546499 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.546670 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.546720 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.546859 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:26:58.546935 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:26:58.547001 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.547046 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.547053 (+ 7us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:26:58.548676 22689 raft_consensus.cc:1261] T 081c77157c454503949d13915f29e97a P 40b09485fa704e9d81a2c7e46034bd82 [term 1 FOLLOWER]: Refusing update from remote peer e01141f462a34bf1a6d4383179152664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:58.548688 22823 raft_consensus.cc:1261] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 1 FOLLOWER]: Refusing update from remote peer e01141f462a34bf1a6d4383179152664: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:58.549014 23027 consensus_queue.cc:1035] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40b09485fa704e9d81a2c7e46034bd82" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 41277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:58.549136 23025 consensus_queue.cc:1035] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49155d1bddf54d48998eefbf164bffe8" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 33817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:58.551919 23032 mvcc.cc:205] Tried to move back new op lower bound from 7332919781572984832 to 7332919781535948800. Current Snapshot: MvccSnapshot[applied={T|T < 7332919781572984832}]
I20260924 15:26:58.552188 23031 mvcc.cc:205] Tried to move back new op lower bound from 7332919781572984832 to 7332919781535948800. Current Snapshot: MvccSnapshot[applied={T|T < 7332919781572984832}]
I20260924 15:26:58.553952 22544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:58.554028 22544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:58.554824 22544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ada860ca3ae43329a61a419c43a1dfb: Sending DeleteTablet for 3 replicas of tablet 081c77157c454503949d13915f29e97a
I20260924 15:26:58.555226 22937 tablet_service.cc:1514] Processing DeleteTablet for tablet 081c77157c454503949d13915f29e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:58 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:55536
I20260924 15:26:58.555231 22803 tablet_service.cc:1514] Processing DeleteTablet for tablet 081c77157c454503949d13915f29e97a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:58 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:39008
I20260924 15:26:58.555413 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 22606
I20260924 15:26:58.556293 23041 tablet_replica.cc:330] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: stopping tablet replica
I20260924 15:26:58.556397 23041 raft_consensus.cc:2229] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:58.556490 23041 raft_consensus.cc:2258] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:58.556912 23041 ts_tablet_manager.cc:1905] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:58.559093 23041 ts_tablet_manager.cc:1918] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:58.559156 23041 log.cc:1196] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/081c77157c454503949d13915f29e97a
I20260924 15:26:58.559437 23041 ts_tablet_manager.cc:1939] T 081c77157c454503949d13915f29e97a P 49155d1bddf54d48998eefbf164bffe8: Deleting consensus metadata
I20260924 15:26:58.559844 22529 catalog_manager.cc:4841] TS 49155d1bddf54d48998eefbf164bffe8 (127.4.78.66:33817): tablet 081c77157c454503949d13915f29e97a (table test-table [id=1968d17a8cf144d4a09b4da4e4922676]) successfully deleted
W20260924 15:26:58.562628 22530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:41277 (error 108)
I20260924 15:26:58.562732 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 22740
I20260924 15:26:58.563072 23011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.562958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:41277 (local address 127.0.0.1:45078)
0924 15:26:58.563006 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.563035 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:41277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 15:26:58.563169 22530 catalog_manager.cc:4584] TS 40b09485fa704e9d81a2c7e46034bd82 (127.4.78.65:41277): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 081c77157c454503949d13915f29e97a: Network error: Client connection negotiation failed: client connection to 127.4.78.65:41277: connect: Connection refused (error 111)
I20260924 15:26:58.564047 23040 tablet_replica.cc:330] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: stopping tablet replica
I20260924 15:26:58.564121 23040 raft_consensus.cc:2229] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:26:58.564213 23040 raft_consensus.cc:2258] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:58.564625 23040 ts_tablet_manager.cc:1905] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:58.566239 23040 ts_tablet_manager.cc:1918] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:58.566313 23040 log.cc:1196] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/081c77157c454503949d13915f29e97a
I20260924 15:26:58.566576 23040 ts_tablet_manager.cc:1939] T 081c77157c454503949d13915f29e97a P e01141f462a34bf1a6d4383179152664: Deleting consensus metadata
I20260924 15:26:58.567265 22529 catalog_manager.cc:4841] TS e01141f462a34bf1a6d4383179152664 (127.4.78.67:41227): tablet 081c77157c454503949d13915f29e97a (table test-table [id=1968d17a8cf144d4a09b4da4e4922676]) successfully deleted
I20260924 15:26:58.568274 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 22874
I20260924 15:26:58.573182 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 22512
2026-09-24T15:26:58Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1874 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T15:26:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:26:58Z Disabled control of system clock
I20260924 15:26:58.601927 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:36729
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:36729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:26:58.686482 23046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:58.686695 23046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:58.686717 23046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:58.687806 23046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:26:58.687840 23046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:58.687852 23046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:58.687865 23046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:26:58.687877 23046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:26:58.689200 23046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:36729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:36729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:58.689399 23046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:58.689630 23046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:58.694358 23052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.694455 23054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.694406 23051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:58.694705 23046 server_base.cc:1034] running on GCE node
I20260924 15:26:58.694942 23046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:58.695230 23046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:58.696455 23046 hybrid_clock.cc:648] HybridClock initialized: now 1790263618696438 us; error 26 us; skew 500 ppm
I20260924 15:26:58.697773 23046 webserver.cc:460] Webserver started at http://127.4.78.126:44209/ using document root <none> and password file <none>
I20260924 15:26:58.698063 23046 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:58.698137 23046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:58.698293 23046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:58.699369 23046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "14bcb038e83226478cf010a5721bf0fd"
server_key_iv: "25eb4ecb2d136c34841d1d228aee02cc"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.699738 23046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "14bcb038e83226478cf010a5721bf0fd"
server_key_iv: "25eb4ecb2d136c34841d1d228aee02cc"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.701036 23046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:26:58.701970 23060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.702140 23046 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:26:58.702243 23046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "14bcb038e83226478cf010a5721bf0fd"
server_key_iv: "25eb4ecb2d136c34841d1d228aee02cc"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.702350 23046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:58.715795 23046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:58.716125 23046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:58.716297 23046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:58.720227 23046 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:36729
I20260924 15:26:58.720325 23112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:36729 every 8 connection(s)
I20260924 15:26:58.720639 23046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:26:58.721599 23113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:58.723933 23113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2: Bootstrap starting.
I20260924 15:26:58.724546 23113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:58.724852 23113 log.cc:824] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:58.725509 23113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2: No bootstrap required, opened a new log
I20260924 15:26:58.726233 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 23046
I20260924 15:26:58.726328 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:26:58.726514 4409 external_mini_cluster.cc:1428] Setting key 3e969a12c2180c6da6da3a8f5831dad7
I20260924 15:26:58.727617 23113 raft_consensus.cc:348] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36729 } }
I20260924 15:26:58.727785 23113 raft_consensus.cc:374] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:58.727828 23113 raft_consensus.cc:729] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e43dcdb3399a430ebe3cb2c866dcd5c2, State: Initialized, Role: FOLLOWER
I20260924 15:26:58.727959 23113 consensus_queue.cc:260] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36729 } }
I20260924 15:26:58.728056 23113 raft_consensus.cc:388] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:26:58.728101 23113 raft_consensus.cc:482] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:26:58.728152 23113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:58.728739 23113 raft_consensus.cc:504] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36729 } }
I20260924 15:26:58.728863 23113 leader_election.cc:302] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e43dcdb3399a430ebe3cb2c866dcd5c2; no voters:
I20260924 15:26:58.729066 23113 leader_election.cc:288] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:26:58.729100 23118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:58.729277 23118 raft_consensus.cc:686] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [term 1 LEADER]: Becoming Leader. State: Replica: e43dcdb3399a430ebe3cb2c866dcd5c2, State: Running, Role: LEADER
I20260924 15:26:58.729346 23113 sys_catalog.cc:564] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:26:58.729360 23118 consensus_queue.cc:237] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36729 } }
I20260924 15:26:58.729678 23118 sys_catalog.cc:455] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e43dcdb3399a430ebe3cb2c866dcd5c2. Latest consensus state: current_term: 1 leader_uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36729 } } }
I20260924 15:26:58.729765 23118 sys_catalog.cc:458] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:58.729688 23120 sys_catalog.cc:455] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e43dcdb3399a430ebe3cb2c866dcd5c2" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 36729 } } }
I20260924 15:26:58.729964 23120 sys_catalog.cc:458] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2 [sys.catalog]: This master's current role is: LEADER
I20260924 15:26:58.730023 23124 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:26:58.730574 23124 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:26:58.732259 23124 catalog_manager.cc:1347] Generated new cluster ID: c3bb8661872d41efa83317904239354d
I20260924 15:26:58.732314 23124 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:26:58.733099 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.727029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56902 (local address 127.4.78.126:36729)
0924 15:26:58.727240 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.727255 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.727279 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:26:58.727602 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.727608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.727662 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.727793 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:58.728004 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.729644 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.730810 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.731409 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.731838 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.732279 (+ 441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:58.732283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:58.732283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:58.732492 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:58.732533 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:58.732547 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:58.732614 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.732714 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.732892 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260924 15:26:58.760390 23124 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:26:58.760946 23124 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:26:58.765079 23124 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2: Generated new TSK 0
I20260924 15:26:58.765304 23124 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:26:58.768537 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36729
--builtin_ntp_servers=127.4.78.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:26:58.858666 23137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:58.858894 23137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:58.858930 23137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:58.860059 23137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:58.860118 23137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:58.860158 23137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:26:58.861490 23137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:58.861752 23137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:58.862052 23137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:58.867486 23143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.867511 23142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.867496 23145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:58.867585 23137 server_base.cc:1034] running on GCE node
I20260924 15:26:58.867866 23137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:58.868119 23137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:58.869288 23137 hybrid_clock.cc:648] HybridClock initialized: now 1790263618869268 us; error 35 us; skew 500 ppm
I20260924 15:26:58.870707 23137 webserver.cc:460] Webserver started at http://127.4.78.65:36583/ using document root <none> and password file <none>
I20260924 15:26:58.871029 23137 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:58.871094 23137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:58.871241 23137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:58.872303 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "3517dcdcb34e4cf88f23be57765a3819"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "91343123ba4ea5980254b01b6731a618"
server_key_iv: "71ebca5e44b49696c47395712eba35ab"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.872666 23137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "3517dcdcb34e4cf88f23be57765a3819"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "91343123ba4ea5980254b01b6731a618"
server_key_iv: "71ebca5e44b49696c47395712eba35ab"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.874064 23137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:26:58.874809 23151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.874984 23137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:58.875082 23137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "3517dcdcb34e4cf88f23be57765a3819"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "91343123ba4ea5980254b01b6731a618"
server_key_iv: "71ebca5e44b49696c47395712eba35ab"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.875167 23137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:58.885442 23137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:58.885730 23137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:58.885934 23137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:58.886158 23137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:58.886482 23137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:58.886541 23137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.886583 23137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:58.886610 23137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.892292 23137 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:46391
I20260924 15:26:58.892349 23264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:46391 every 8 connection(s)
I20260924 15:26:58.892817 23137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:26:58.893980 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 23137
I20260924 15:26:58.894093 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:26:58.894271 4409 external_mini_cluster.cc:1428] Setting key bb1e1b0990648fb2287e9a314d1b8c32
I20260924 15:26:58.895288 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36729
--builtin_ntp_servers=127.4.78.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:26:58.899650 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.893647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:38795 (local address 127.4.78.126:36729)
0924 15:26:58.893745 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:58.893748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:58.894028 (+ 280us) server_negotiation.cc:407] Connection header received
0924 15:26:58.894696 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:58.894697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:58.894711 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:58.894788 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:58.895640 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.896452 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.898126 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.898321 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.898384 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.899384 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:58.899386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:58.899386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:58.899390 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:58.899405 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:58.899409 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:58.899447 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:58.899565 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 15:26:58.899591 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:26:58.899966 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:58.893659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:36729 (local address 127.4.78.65:38795)
0924 15:26:58.893886 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:58.893927 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:58.894605 (+ 678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:58.894843 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:58.894847 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:58.894855 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:58.895543 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.895549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.896643 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.896644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.898046 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:58.898049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:58.898419 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:58.898421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:58.898587 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:58.899180 (+ 593us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:26:58.899181 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:58.899223 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:26:58.899256 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:58.899474 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:58.899481 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:58.899484 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:26:58.899567 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:26:58.899691 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":110,"threads_started":1}
I20260924 15:26:58.900565 23265 heartbeater.cc:347] Connected to a master server at 127.4.78.126:36729
I20260924 15:26:58.900671 23265 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:58.900892 23265 heartbeater.cc:511] Master 127.4.78.126:36729 requested a full tablet report, sending...
I20260924 15:26:58.901443 23077 ts_manager.cc:194] Registered new tserver with Master: 3517dcdcb34e4cf88f23be57765a3819 (127.4.78.65:46391)
I20260924 15:26:58.902132 23077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.65:38795
W20260924 15:26:58.977190 23268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:58.977375 23268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:58.977396 23268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:58.978713 23268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:58.978760 23268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:58.978830 23268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:26:58.980185 23268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:58.980420 23268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:58.980722 23268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:58.985582 23273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.985651 23276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:58.985651 23274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:58.985757 23268 server_base.cc:1034] running on GCE node
I20260924 15:26:58.986032 23268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:58.986301 23268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:58.987463 23268 hybrid_clock.cc:648] HybridClock initialized: now 1790263618987454 us; error 38 us; skew 500 ppm
I20260924 15:26:58.988670 23268 webserver.cc:460] Webserver started at http://127.4.78.66:44063/ using document root <none> and password file <none>
I20260924 15:26:58.988941 23268 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:58.989014 23268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:58.989181 23268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:58.990296 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "9a9a3866aa0f4609a6b55c824a71ab1e"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "3b39ee9291ab61dc63268379c91c40f8"
server_key_iv: "c9cafcc2c3f1120f4c4a4814dfb7be50"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.990649 23268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "9a9a3866aa0f4609a6b55c824a71ab1e"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "3b39ee9291ab61dc63268379c91c40f8"
server_key_iv: "c9cafcc2c3f1120f4c4a4814dfb7be50"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.991940 23268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:26:58.992666 23282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:58.992839 23268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:26:58.992937 23268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "9a9a3866aa0f4609a6b55c824a71ab1e"
format_stamp: "Formatted at 2026-09-24 15:26:58 on dist-test-slave-l766"
server_key: "3b39ee9291ab61dc63268379c91c40f8"
server_key_iv: "c9cafcc2c3f1120f4c4a4814dfb7be50"
server_key_version: "encryptionkey@0"
I20260924 15:26:58.993019 23268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:59.018599 23268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:59.018937 23268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:59.019089 23268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:59.019317 23268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:59.019656 23268 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:59.019711 23268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:59.019752 23268 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:59.019779 23268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:59.025743 23268 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:35299
I20260924 15:26:59.025792 23396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:35299 every 8 connection(s)
I20260924 15:26:59.026126 23268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:26:59.031025 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 23268
I20260924 15:26:59.031132 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:26:59.031337 4409 external_mini_cluster.cc:1428] Setting key 1113c4b8bb814bf6490ca953e3366ad2
I20260924 15:26:59.031785 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.026862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:45701 (local address 127.4.78.126:36729)
0924 15:26:59.026979 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.026981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.027158 (+ 177us) server_negotiation.cc:407] Connection header received
0924 15:26:59.027713 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.027714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.027728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.027780 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:59.028476 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.029098 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.030232 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.030444 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.030532 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.031485 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:59.031487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:59.031488 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:59.031494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:59.031510 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:59.031517 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:59.031569 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.031684 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.031716 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 15:26:59.032029 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.026845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:36729 (local address 127.4.78.66:45701)
0924 15:26:59.027027 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:59.027074 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:59.027592 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:59.027855 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:59.027859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:59.027875 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:59.028376 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.028381 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.029211 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.029212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.030136 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.030138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.030542 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.030543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.030690 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.031284 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:26:59.031285 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:59.031321 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:26:59.031344 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:59.031593 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:59.031599 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:59.031602 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:26:59.031673 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:26:59.031802 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260924 15:26:59.032478 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36729
--builtin_ntp_servers=127.4.78.84:42577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:26:59.032492 23397 heartbeater.cc:347] Connected to a master server at 127.4.78.126:36729
I20260924 15:26:59.032601 23397 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:59.032788 23397 heartbeater.cc:511] Master 127.4.78.126:36729 requested a full tablet report, sending...
I20260924 15:26:59.033185 23077 ts_manager.cc:194] Registered new tserver with Master: 9a9a3866aa0f4609a6b55c824a71ab1e (127.4.78.66:35299)
I20260924 15:26:59.033690 23077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.66:45701
W20260924 15:26:59.112452 23400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:26:59.112644 23400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:26:59.112664 23400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:26:59.113713 23400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:26:59.113745 23400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:26:59.113770 23400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:26:59.115084 23400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:42577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:36729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:26:59.115255 23400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:26:59.115479 23400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:26:59.120219 23405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:59.120374 23400 server_base.cc:1034] running on GCE node
W20260924 15:26:59.120221 23406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:26:59.120203 23408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:26:59.120666 23400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:26:59.120915 23400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:26:59.122072 23400 hybrid_clock.cc:648] HybridClock initialized: now 1790263619122072 us; error 38 us; skew 500 ppm
I20260924 15:26:59.123376 23400 webserver.cc:460] Webserver started at http://127.4.78.67:34353/ using document root <none> and password file <none>
I20260924 15:26:59.123607 23400 fs_manager.cc:359] Metadata directory not provided
I20260924 15:26:59.123678 23400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:26:59.123838 23400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:26:59.124845 23400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0"
format_stamp: "Formatted at 2026-09-24 15:26:59 on dist-test-slave-l766"
server_key: "b30b008cf16de250c14a0fcc3d16febd"
server_key_iv: "e03b2709d94ece5c5a6e68f9fd1720a6"
server_key_version: "encryptionkey@0"
I20260924 15:26:59.125191 23400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0"
format_stamp: "Formatted at 2026-09-24 15:26:59 on dist-test-slave-l766"
server_key: "b30b008cf16de250c14a0fcc3d16febd"
server_key_iv: "e03b2709d94ece5c5a6e68f9fd1720a6"
server_key_version: "encryptionkey@0"
I20260924 15:26:59.126518 23400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:26:59.127279 23414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:59.127445 23400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:26:59.127540 23400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0"
format_stamp: "Formatted at 2026-09-24 15:26:59 on dist-test-slave-l766"
server_key: "b30b008cf16de250c14a0fcc3d16febd"
server_key_iv: "e03b2709d94ece5c5a6e68f9fd1720a6"
server_key_version: "encryptionkey@0"
I20260924 15:26:59.127624 23400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:26:59.152094 23400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:26:59.152422 23400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:26:59.152580 23400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:26:59.152805 23400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:26:59.153146 23400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:26:59.153201 23400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:59.153244 23400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:26:59.153270 23400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:26:59.158932 23400 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:42607
I20260924 15:26:59.158977 23527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:42607 every 8 connection(s)
I20260924 15:26:59.159296 23400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:26:59.164462 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.160019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:39079 (local address 127.4.78.126:36729)
0924 15:26:59.160082 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.160084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.160337 (+ 253us) server_negotiation.cc:407] Connection header received
0924 15:26:59.160861 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.160862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.160877 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.160929 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:59.161597 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.162174 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.163131 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.163335 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.163424 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.164148 (+ 724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:59.164150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:59.164150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:59.164155 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:59.164169 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:59.164173 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:59.164239 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.164354 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.164385 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:26:59.164682 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.160021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:36729 (local address 127.4.78.67:39079)
0924 15:26:59.160201 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:59.160233 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:59.160747 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:59.161010 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:59.161014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:59.161022 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:26:59.161502 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.161506 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.162268 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.162269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.163035 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.163037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.163428 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.163429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.163555 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.164003 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:26:59.164004 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:26:59.164031 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:26:59.164051 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:26:59.164239 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:26:59.164245 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:26:59.164247 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:26:59.164357 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:26:59.164456 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260924 15:26:59.165158 23528 heartbeater.cc:347] Connected to a master server at 127.4.78.126:36729
I20260924 15:26:59.165246 23528 heartbeater.cc:464] Registering TS with master...
I20260924 15:26:59.165393 23528 heartbeater.cc:511] Master 127.4.78.126:36729 requested a full tablet report, sending...
I20260924 15:26:59.165717 23077 ts_manager.cc:194] Registered new tserver with Master: 8a9fcce47d6d4e98a2cf61dd0e28d2d0 (127.4.78.67:42607)
I20260924 15:26:59.166057 23077 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.67:39079
I20260924 15:26:59.167564 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 23400
I20260924 15:26:59.167645 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:26:59.167784 4409 external_mini_cluster.cc:1428] Setting key 99212aa6db47c87aeb6025e6173cd497
I20260924 15:26:59.168614 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:26:59.172619 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.169907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.4.78.126:36729)
0924 15:26:59.169980 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.169982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.170071 (+ 89us) server_negotiation.cc:407] Connection header received
0924 15:26:59.170092 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.170093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.170103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.170145 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:26:59.170312 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.170760 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.171487 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.171653 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.171701 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.172340 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:26:59.172341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:26:59.172341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:26:59.172344 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:26:59.172355 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:26:59.172358 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:26:59.172427 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.172485 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.172525 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:26:59.174341 23077 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56908:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:26:59.175892 23077 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:26:59.181594 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.177937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:42607 (local address 127.0.0.1:59258)
0924 15:26:59.178249 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:59.178282 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:59.178354 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:59.178583 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:59.178585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:59.178588 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:59.178651 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.178653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.179679 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.179680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.180467 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.180469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.181305 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.181306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.181393 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.181406 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:26:59.181471 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 15:26:59.181522 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":78,"threads_started":1}
I20260924 15:26:59.181890 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.178000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59258 (local address 127.4.78.67:42607)
0924 15:26:59.178254 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.178258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.178349 (+ 91us) server_negotiation.cc:407] Connection header received
0924 15:26:59.178430 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.178433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.178446 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.178517 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:59.178728 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.179596 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.180559 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.181204 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.181301 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.181386 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.181759 (+ 373us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.181795 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":74,"threads_started":1}
I20260924 15:26:59.182622 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.178128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:35299 (local address 127.0.0.1:51910)
0924 15:26:59.178275 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:59.178284 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:59.178349 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:59.178686 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:59.178687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:59.178689 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:59.178748 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.178750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.180101 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.180102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.181338 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.181340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.182389 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.182390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.182450 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.182454 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:59.182513 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:26:59.182556 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":44,"thread_start_us":78,"threads_started":1}
I20260924 15:26:59.182538 23462 tablet_service.cc:1467] Processing CreateTablet for tablet 907d338e685f4614a525c4f806e5e557 (DEFAULT_TABLE table=test-table [id=9dca6cfa4bb94ed1ae8b69f5bb2a7acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:59.182677 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.178344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.4.78.66:35299)
0924 15:26:59.178531 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.178535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.178545 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:26:59.178582 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.178584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.178598 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.178675 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:59.178837 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.180000 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.181438 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.182205 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.182274 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.182352 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.182546 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.182595 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":108,"threads_started":1}
I20260924 15:26:59.182747 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.178003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.4.78.65:46391)
0924 15:26:59.178162 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.178166 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.178487 (+ 321us) server_negotiation.cc:407] Connection header received
0924 15:26:59.178609 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.178612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.178629 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.178721 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:59.178909 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.180008 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.181457 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.182254 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.182326 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.182417 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.182619 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.182661 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260924 15:26:59.182891 23462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907d338e685f4614a525c4f806e5e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:59.182925 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.177925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:46391 (local address 127.0.0.1:46666)
0924 15:26:59.178178 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:59.178435 (+ 257us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:59.178515 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:59.178789 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:59.178790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:59.178792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:59.178826 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.178827 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.180109 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.180110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.181340 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.181341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.182448 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.182449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.182522 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.182526 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:59.182813 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 15:26:59.182867 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":85,"thread_start_us":63,"threads_started":1}
I20260924 15:26:59.183516 23199 tablet_service.cc:1467] Processing CreateTablet for tablet 907d338e685f4614a525c4f806e5e557 (DEFAULT_TABLE table=test-table [id=9dca6cfa4bb94ed1ae8b69f5bb2a7acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:59.183743 23199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907d338e685f4614a525c4f806e5e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:59.185462 23543 tablet_bootstrap.cc:492] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: Bootstrap starting.
I20260924 15:26:59.186089 23331 tablet_service.cc:1467] Processing CreateTablet for tablet 907d338e685f4614a525c4f806e5e557 (DEFAULT_TABLE table=test-table [id=9dca6cfa4bb94ed1ae8b69f5bb2a7acd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:26:59.186393 23544 tablet_bootstrap.cc:492] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819: Bootstrap starting.
I20260924 15:26:59.186434 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907d338e685f4614a525c4f806e5e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:26:59.186492 23543 tablet_bootstrap.cc:654] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:59.186821 23543 log.cc:824] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:59.187014 23544 tablet_bootstrap.cc:654] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:59.187356 23544 log.cc:824] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:59.187616 23543 tablet_bootstrap.cc:492] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: No bootstrap required, opened a new log
I20260924 15:26:59.187695 23543 ts_tablet_manager.cc:1397] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:26:59.188014 23544 tablet_bootstrap.cc:492] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819: No bootstrap required, opened a new log
I20260924 15:26:59.188095 23544 ts_tablet_manager.cc:1397] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:26:59.189885 23547 tablet_bootstrap.cc:492] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: Bootstrap starting.
I20260924 15:26:59.190258 23544 raft_consensus.cc:348] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.190415 23544 raft_consensus.cc:374] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:59.190457 23544 raft_consensus.cc:729] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3517dcdcb34e4cf88f23be57765a3819, State: Initialized, Role: FOLLOWER
I20260924 15:26:59.190531 23547 tablet_bootstrap.cc:654] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: Neither blocks nor log segments found. Creating new log.
I20260924 15:26:59.190629 23544 consensus_queue.cc:260] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.190693 23543 raft_consensus.cc:348] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.190836 23547 log.cc:824] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:26:59.190869 23543 raft_consensus.cc:374] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:59.190910 23543 raft_consensus.cc:729] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a9fcce47d6d4e98a2cf61dd0e28d2d0, State: Initialized, Role: FOLLOWER
I20260924 15:26:59.190910 23544 ts_tablet_manager.cc:1428] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:26:59.191057 23265 heartbeater.cc:503] Master 127.4.78.126:36729 was elected leader, sending a full tablet report...
I20260924 15:26:59.191289 23543 consensus_queue.cc:260] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.191613 23543 ts_tablet_manager.cc:1428] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260924 15:26:59.191634 23528 heartbeater.cc:503] Master 127.4.78.126:36729 was elected leader, sending a full tablet report...
I20260924 15:26:59.191784 23548 raft_consensus.cc:482] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:26:59.191860 23548 raft_consensus.cc:504] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.192168 23548 leader_election.cc:288] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a9fcce47d6d4e98a2cf61dd0e28d2d0 (127.4.78.67:42607), 9a9a3866aa0f4609a6b55c824a71ab1e (127.4.78.66:35299)
I20260924 15:26:59.192965 23547 tablet_bootstrap.cc:492] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: No bootstrap required, opened a new log
I20260924 15:26:59.193035 23547 ts_tablet_manager.cc:1397] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:26:59.195611 23547 raft_consensus.cc:348] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.195834 23547 raft_consensus.cc:374] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:26:59.195886 23547 raft_consensus.cc:729] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a9a3866aa0f4609a6b55c824a71ab1e, State: Initialized, Role: FOLLOWER
I20260924 15:26:59.195947 23267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.192334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:42607 (local address 127.4.78.65:36055)
0924 15:26:59.192375 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:59.192386 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:59.192428 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:59.192590 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:59.192591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:59.192593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:59.192652 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.192653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.193513 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.193514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.194975 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.194978 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.195706 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.195707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.195775 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.195779 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:59.195836 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:26:59.195880 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 15:26:59.196000 23547 consensus_queue.cc:260] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.196156 23542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.192578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:48117 (local address 127.4.78.66:35299)
0924 15:26:59.192611 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.192613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.192796 (+ 183us) server_negotiation.cc:407] Connection header received
0924 15:26:59.192892 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.192893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.192901 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.192930 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:59.193157 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.193966 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.195288 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.195799 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.195864 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.195872 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.196058 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.196103 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:26:59.196189 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.192363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:36055 (local address 127.4.78.67:42607)
0924 15:26:59.192401 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.192403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.192416 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:26:59.192482 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.192483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.192493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.192545 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:26:59.192722 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.193415 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.195074 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.195623 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.195700 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.195719 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.196098 (+ 379us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.196133 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:26:59.196286 23547 ts_tablet_manager.cc:1428] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260924 15:26:59.196349 23552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.192429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:35299 (local address 127.4.78.65:48117)
0924 15:26:59.192733 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:59.192744 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:26:59.192823 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:26:59.193003 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:26:59.193004 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:26:59.193007 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:26:59.193075 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.193077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.194060 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.194061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.195205 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:26:59.195207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.195909 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.195910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:26:59.195970 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.195974 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:26:59.196229 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 15:26:59.196276 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":76,"threads_started":1}
I20260924 15:26:59.196398 23397 heartbeater.cc:503] Master 127.4.78.126:36729 was elected leader, sending a full tablet report...
I20260924 15:26:59.196405 23482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907d338e685f4614a525c4f806e5e557" candidate_uuid: "3517dcdcb34e4cf88f23be57765a3819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" is_pre_election: true
I20260924 15:26:59.196530 23482 raft_consensus.cc:2454] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3517dcdcb34e4cf88f23be57765a3819 in term 0.
I20260924 15:26:59.196832 23154 leader_election.cc:302] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517dcdcb34e4cf88f23be57765a3819, 8a9fcce47d6d4e98a2cf61dd0e28d2d0; no voters:
I20260924 15:26:59.196990 23548 raft_consensus.cc:2781] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:26:59.197065 23548 raft_consensus.cc:482] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:26:59.197105 23548 raft_consensus.cc:3037] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:59.197170 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907d338e685f4614a525c4f806e5e557" candidate_uuid: "3517dcdcb34e4cf88f23be57765a3819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" is_pre_election: true
I20260924 15:26:59.197299 23351 raft_consensus.cc:2454] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3517dcdcb34e4cf88f23be57765a3819 in term 0.
I20260924 15:26:59.197882 23548 raft_consensus.cc:504] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.198004 23548 leader_election.cc:288] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [CANDIDATE]: Term 1 election: Requested vote from peers 8a9fcce47d6d4e98a2cf61dd0e28d2d0 (127.4.78.67:42607), 9a9a3866aa0f4609a6b55c824a71ab1e (127.4.78.66:35299)
I20260924 15:26:59.198146 23482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907d338e685f4614a525c4f806e5e557" candidate_uuid: "3517dcdcb34e4cf88f23be57765a3819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0"
I20260924 15:26:59.198187 23351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907d338e685f4614a525c4f806e5e557" candidate_uuid: "3517dcdcb34e4cf88f23be57765a3819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e"
I20260924 15:26:59.198226 23482 raft_consensus.cc:3037] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:59.198241 23351 raft_consensus.cc:3037] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:26:59.199041 23351 raft_consensus.cc:2454] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3517dcdcb34e4cf88f23be57765a3819 in term 1.
I20260924 15:26:59.199041 23482 raft_consensus.cc:2454] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3517dcdcb34e4cf88f23be57765a3819 in term 1.
I20260924 15:26:59.199210 23154 leader_election.cc:302] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3517dcdcb34e4cf88f23be57765a3819, 9a9a3866aa0f4609a6b55c824a71ab1e; no voters:
I20260924 15:26:59.199347 23548 raft_consensus.cc:2781] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:26:59.199440 23548 raft_consensus.cc:686] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [term 1 LEADER]: Becoming Leader. State: Replica: 3517dcdcb34e4cf88f23be57765a3819, State: Running, Role: LEADER
I20260924 15:26:59.199515 23548 consensus_queue.cc:237] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } }
I20260924 15:26:59.200258 23077 catalog_manager.cc:5495] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3517dcdcb34e4cf88f23be57765a3819 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "3517dcdcb34e4cf88f23be57765a3819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3517dcdcb34e4cf88f23be57765a3819" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 46391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 } health_report { overall_health: UNKNOWN } } }
I20260924 15:26:59.211535 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.208024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.4.78.65:46391)
0924 15:26:59.208080 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:26:59.208082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:26:59.208091 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:26:59.208131 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:26:59.208132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:26:59.208146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:26:59.208203 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:26:59.208366 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.209084 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.210447 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:26:59.210781 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:26:59.210866 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:26:59.211069 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:26:59.211237 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:26:59.211362 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 15:26:59.211427 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:26:59.211439 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:26:59.214143 23351 raft_consensus.cc:1261] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 1 FOLLOWER]: Refusing update from remote peer 3517dcdcb34e4cf88f23be57765a3819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:59.214226 23482 raft_consensus.cc:1261] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 1 FOLLOWER]: Refusing update from remote peer 3517dcdcb34e4cf88f23be57765a3819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:26:59.214602 23548 consensus_queue.cc:1035] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a9fcce47d6d4e98a2cf61dd0e28d2d0" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:59.214810 23548 consensus_queue.cc:1035] T 907d338e685f4614a525c4f806e5e557 P 3517dcdcb34e4cf88f23be57765a3819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a9a3866aa0f4609a6b55c824a71ab1e" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 35299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:26:59.217398 23558 mvcc.cc:205] Tried to move back new op lower bound from 7332919784298594304 to 7332919784242012160. Current Snapshot: MvccSnapshot[applied={T|T < 7332919784298594304}]
I20260924 15:26:59.217456 23556 mvcc.cc:205] Tried to move back new op lower bound from 7332919784298594304 to 7332919784242012160. Current Snapshot: MvccSnapshot[applied={T|T < 7332919784298594304 or (T in {7332919784298594304})}]
I20260924 15:26:59.219260 23077 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:59.219348 23077 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56908:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:26:59.220036 23077 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e43dcdb3399a430ebe3cb2c866dcd5c2: Sending DeleteTablet for 3 replicas of tablet 907d338e685f4614a525c4f806e5e557
I20260924 15:26:59.220356 23331 tablet_service.cc:1514] Processing DeleteTablet for tablet 907d338e685f4614a525c4f806e5e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:59 UTC) from {username='slave'} at 127.0.0.1:51910
I20260924 15:26:59.220379 23199 tablet_service.cc:1514] Processing DeleteTablet for tablet 907d338e685f4614a525c4f806e5e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:59 UTC) from {username='slave'} at 127.0.0.1:46666
I20260924 15:26:59.220422 23462 tablet_service.cc:1514] Processing DeleteTablet for tablet 907d338e685f4614a525c4f806e5e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:26:59 UTC) from {username='slave'} at 127.0.0.1:59258
I20260924 15:26:59.220625 23568 tablet_replica.cc:330] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: stopping tablet replica
I20260924 15:26:59.220773 23568 raft_consensus.cc:2229] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:59.220779 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 23137
I20260924 15:26:59.220935 23568 raft_consensus.cc:2258] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:59.222599 23566 tablet_replica.cc:330] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: stopping tablet replica
I20260924 15:26:59.222708 23566 raft_consensus.cc:2229] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:26:59.222785 23568 ts_tablet_manager.cc:1905] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:59.222807 23566 raft_consensus.cc:2258] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:26:59.223178 23566 ts_tablet_manager.cc:1905] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:26:59.224766 23566 ts_tablet_manager.cc:1918] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:59.224774 23568 ts_tablet_manager.cc:1918] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:26:59.224838 23568 log.cc:1196] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/907d338e685f4614a525c4f806e5e557
I20260924 15:26:59.224862 23566 log.cc:1196] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/907d338e685f4614a525c4f806e5e557
I20260924 15:26:59.225087 23568 ts_tablet_manager.cc:1939] T 907d338e685f4614a525c4f806e5e557 P 8a9fcce47d6d4e98a2cf61dd0e28d2d0: Deleting consensus metadata
I20260924 15:26:59.225108 23566 ts_tablet_manager.cc:1939] T 907d338e685f4614a525c4f806e5e557 P 9a9a3866aa0f4609a6b55c824a71ab1e: Deleting consensus metadata
I20260924 15:26:59.225481 23063 catalog_manager.cc:4841] TS 8a9fcce47d6d4e98a2cf61dd0e28d2d0 (127.4.78.67:42607): tablet 907d338e685f4614a525c4f806e5e557 (table test-table [id=9dca6cfa4bb94ed1ae8b69f5bb2a7acd]) successfully deleted
I20260924 15:26:59.225595 23063 catalog_manager.cc:4841] TS 9a9a3866aa0f4609a6b55c824a71ab1e (127.4.78.66:35299): tablet 907d338e685f4614a525c4f806e5e557 (table test-table [id=9dca6cfa4bb94ed1ae8b69f5bb2a7acd]) successfully deleted
W20260924 15:26:59.227217 23061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:46391 (error 108)
I20260924 15:26:59.227454 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 23268
I20260924 15:26:59.228041 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:26:59.227532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:46391 (local address 127.0.0.1:46676)
0924 15:26:59.227964 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0924 15:26:59.228004 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:46391: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411}
W20260924 15:26:59.228235 23061 catalog_manager.cc:4584] TS 3517dcdcb34e4cf88f23be57765a3819 (127.4.78.65:46391): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 907d338e685f4614a525c4f806e5e557: Network error: Client connection negotiation failed: client connection to 127.4.78.65:46391: connect: Connection refused (error 111)
I20260924 15:26:59.233023 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 23400
I20260924 15:26:59.237250 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 23046
2026-09-24T15:26:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (661 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:26:59 dist-test-slave-l766 krb5kdc[23572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:26:59 dist-test-slave-l766 krb5kdc[23572](info): set up 2 sockets
Sep 24 15:26:59 dist-test-slave-l766 krb5kdc[23572](info): commencing operation
krb5kdc: starting...
W20260924 15:27:00.280692 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:00 dist-test-slave-l766 krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:00.396798 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:40757
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:40757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:00.476774 23585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:00.476997 23585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:00.477035 23585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:00.478226 23585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:00.478292 23585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:00.478320 23585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:00.478353 23585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:00.478376 23585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:00.479797 23585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:40757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:40757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:00.480064 23585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:00.480293 23585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:00.484990 23590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:00.484984 23591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:00.484983 23593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:00.485235 23585 server_base.cc:1034] running on GCE node
I20260924 15:27:00.485457 23585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:00.485680 23585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:00.486861 23585 hybrid_clock.cc:648] HybridClock initialized: now 1790263620486849 us; error 45 us; skew 500 ppm
Sep 24 15:27:00 dist-test-slave-l766 krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263620, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:00.498425 23585 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:00.498893 23585 webserver.cc:460] Webserver started at http://127.4.78.126:38171/ using document root <none> and password file <none>
I20260924 15:27:00.499163 23585 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:00.499380 23585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:00.499565 23585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:00.500685 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "d5d097ba87b34331832dfeabee212e0c"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "1e22ae2d78deca68572d84ef3cb13261"
server_key_iv: "596e0bdd5b088b0f90c00fe328a449b5"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.501067 23585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "d5d097ba87b34331832dfeabee212e0c"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "1e22ae2d78deca68572d84ef3cb13261"
server_key_iv: "596e0bdd5b088b0f90c00fe328a449b5"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.502563 23585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:00.503425 23600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:00.503633 23585 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:00.503727 23585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "d5d097ba87b34331832dfeabee212e0c"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "1e22ae2d78deca68572d84ef3cb13261"
server_key_iv: "596e0bdd5b088b0f90c00fe328a449b5"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.503822 23585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:00.515899 23585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:00.517720 23585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:00.517907 23585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:00.521991 23585 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:40757
I20260924 15:27:00.522023 23653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:40757 every 8 connection(s)
I20260924 15:27:00.522369 23585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:00.523209 23654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:00.525435 23654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c: Bootstrap starting.
I20260924 15:27:00.526060 23654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:00.526346 23654 log.cc:824] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:00.527019 23654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c: No bootstrap required, opened a new log
I20260924 15:27:00.528563 23654 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d097ba87b34331832dfeabee212e0c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40757 } }
I20260924 15:27:00.528705 23654 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:00.528762 23654 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d097ba87b34331832dfeabee212e0c, State: Initialized, Role: FOLLOWER
I20260924 15:27:00.528924 23654 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d097ba87b34331832dfeabee212e0c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40757 } }
I20260924 15:27:00.529008 23654 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:00.529059 23654 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:00.529109 23654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:00.529636 23654 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d097ba87b34331832dfeabee212e0c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40757 } }
I20260924 15:27:00.529757 23654 leader_election.cc:302] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5d097ba87b34331832dfeabee212e0c; no voters:
I20260924 15:27:00.529950 23654 leader_election.cc:288] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:00.529984 23657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:00.530129 23657 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [term 1 LEADER]: Becoming Leader. State: Replica: d5d097ba87b34331832dfeabee212e0c, State: Running, Role: LEADER
I20260924 15:27:00.530207 23654 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:00.530217 23657 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d097ba87b34331832dfeabee212e0c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40757 } }
I20260924 15:27:00.530663 23658 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5d097ba87b34331832dfeabee212e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d097ba87b34331832dfeabee212e0c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40757 } } }
I20260924 15:27:00.530747 23658 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:00.530916 23659 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5d097ba87b34331832dfeabee212e0c. Latest consensus state: current_term: 1 leader_uuid: "d5d097ba87b34331832dfeabee212e0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5d097ba87b34331832dfeabee212e0c" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 40757 } } }
I20260924 15:27:00.530965 23665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:00.531075 23659 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:00.531693 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 23585
I20260924 15:27:00.531788 23665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:00.531811 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:00.531994 4409 external_mini_cluster.cc:1428] Setting key 3408840752f4e0427d07aec5169b184b
I20260924 15:27:00.533524 23665 catalog_manager.cc:1347] Generated new cluster ID: 4de20bbb77e64de1996a46f632b95e83
I20260924 15:27:00.533595 23665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:00 dist-test-slave-l766 krb5kdc[23572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:00.544960 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:00.532590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.4.78.126:40757)
0924 15:27:00.532739 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:00.532748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:00.534087 (+ 1339us) server_negotiation.cc:407] Connection header received
0924 15:27:00.534229 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:00.534236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:00.534290 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:00.534444 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:00.535136 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.536578 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.537638 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.538356 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.538494 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:00.541357 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:00.541362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:00.541362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:00.541374 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:00.543309 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:00.543704 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:00.543706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:00.543706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:00.543812 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:00.544024 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:00.544025 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:00.544025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:00.544323 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:00.544452 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:00.544582 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 15:27:00.544714 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
I20260924 15:27:00.546027 23665 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:00.546507 23665 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:00.556345 23665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c: Generated new TSK 0
I20260924 15:27:00.556510 23665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.78.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
Entry for principal kudu/127.4.78.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab.
I20260924 15:27:00.584293 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40757
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:00.668712 23680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:00.668970 23680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:00.669010 23680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:00.670229 23680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:00.670300 23680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:00.670346 23680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:27:00.671810 23680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.65.keytab
--principal=kudu/127.4.78.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:00.672094 23680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:00.672340 23680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:00.677445 23685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:00.677469 23686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:00.677487 23688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:00.677765 23680 server_base.cc:1034] running on GCE node
I20260924 15:27:00.678026 23680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:00.678283 23680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:00.679473 23680 hybrid_clock.cc:648] HybridClock initialized: now 1790263620679439 us; error 48 us; skew 500 ppm
Sep 24 15:27:00 dist-test-slave-l766 krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263620, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:00.682080 23680 init.cc:377] Logged in from keytab as kudu/127.4.78.65@KRBTEST.COM (short username kudu)
I20260924 15:27:00.682577 23680 webserver.cc:460] Webserver started at http://127.4.78.65:41031/ using document root <none> and password file <none>
I20260924 15:27:00.682817 23680 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:00.682862 23680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:00.682951 23680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:00.683969 23680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "13d042aa6ae94bdfb856b4a7bab4ad18"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "6705029616b66553c50e1c78563b02fa"
server_key_iv: "5f5355cebc2c2272be7066034b7d1dba"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.684264 23680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "13d042aa6ae94bdfb856b4a7bab4ad18"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "6705029616b66553c50e1c78563b02fa"
server_key_iv: "5f5355cebc2c2272be7066034b7d1dba"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.685521 23680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:00.686416 23695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:00.686627 23680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:27:00.686731 23680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "13d042aa6ae94bdfb856b4a7bab4ad18"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "6705029616b66553c50e1c78563b02fa"
server_key_iv: "5f5355cebc2c2272be7066034b7d1dba"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.686813 23680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:00.695250 23680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:00.697163 23680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:00.697266 23680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:00.697544 23680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:00.697906 23680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:00.697944 23680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:00.697965 23680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:00.697978 23680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:00.703977 23680 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:40363
I20260924 15:27:00.704028 23809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:40363 every 8 connection(s)
I20260924 15:27:00.704341 23680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:27:00 dist-test-slave-l766 krb5kdc[23572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263620, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.65@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:00.713227 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 23680
I20260924 15:27:00.713347 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:00.713541 4409 external_mini_cluster.cc:1428] Setting key 4d2f28bc3c9c4f79ef2436527c1128d0
I20260924 15:27:00.713903 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:00.705074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:50149 (local address 127.4.78.126:40757)
0924 15:27:00.705147 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:00.705149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:00.705363 (+ 214us) server_negotiation.cc:407] Connection header received
0924 15:27:00.705873 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:00.705874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:00.705890 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:00.705957 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:00.706909 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.707548 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.708518 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.708725 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.708780 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:00.710823 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:00.710824 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:00.710824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:00.710833 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:00.712229 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:00.712715 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:00.712716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:00.712716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:00.712780 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:00.713082 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:00.713083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:00.713084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:00.713236 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:00.713330 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:00.713660 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 15:27:00.713752 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:27:00.714046 23812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:00.705054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40757 (local address 127.4.78.65:50149)
0924 15:27:00.705221 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:00.705263 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:00.705739 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:00.706015 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:00.706018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:00.706630 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:00.706821 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:00.706825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.707643 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.707645 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:00.708440 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:00.708442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.708798 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.708799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:00.708910 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:00.709364 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:00.709373 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:00.710738 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:00.712332 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:00.712339 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:00.712341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:00.712650 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:00.712869 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:00.712870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:00.712870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:00.712975 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:00.713342 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:00.713349 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:00.713501 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 15:27:00.713620 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 15:27:00.713798 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260924 15:27:00.714738 23810 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40757
I20260924 15:27:00.714852 23810 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:00.715032 23810 heartbeater.cc:511] Master 127.4.78.126:40757 requested a full tablet report, sending...
I20260924 15:27:00.715652 23617 ts_manager.cc:194] Registered new tserver with Master: 13d042aa6ae94bdfb856b4a7bab4ad18 (127.4.78.65:40363)
I20260924 15:27:00.716353 23617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.65@KRBTEST.COM'} at 127.4.78.65:50149
WARNING: no policy specified for kudu/127.4.78.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
Entry for principal kudu/127.4.78.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab.
I20260924 15:27:00.739112 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40757
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:00.820700 23815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:00.820958 23815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:00.820998 23815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:00.822192 23815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:00.822271 23815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:00.822324 23815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:27:00.823797 23815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.66.keytab
--principal=kudu/127.4.78.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:00.824080 23815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:00.824334 23815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:00.829341 23820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:00.829394 23823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:00.829342 23821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:00.829490 23815 server_base.cc:1034] running on GCE node
I20260924 15:27:00.829921 23815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:00.830166 23815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:00.831372 23815 hybrid_clock.cc:648] HybridClock initialized: now 1790263620831355 us; error 27 us; skew 500 ppm
Sep 24 15:27:00 dist-test-slave-l766 krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263620, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:00.834079 23815 init.cc:377] Logged in from keytab as kudu/127.4.78.66@KRBTEST.COM (short username kudu)
I20260924 15:27:00.834580 23815 webserver.cc:460] Webserver started at http://127.4.78.66:46059/ using document root <none> and password file <none>
I20260924 15:27:00.834837 23815 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:00.834906 23815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:00.835063 23815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:00.836120 23815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "0a212890a18e41ea89370a12b311a5a0"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "1e5c9a5574058139d88cd02df3a1565a"
server_key_iv: "59a19f456179c81b83b57f4882555853"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.836473 23815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "0a212890a18e41ea89370a12b311a5a0"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "1e5c9a5574058139d88cd02df3a1565a"
server_key_iv: "59a19f456179c81b83b57f4882555853"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.837792 23815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:00.838644 23830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:00.838908 23815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:00.839018 23815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "0a212890a18e41ea89370a12b311a5a0"
format_stamp: "Formatted at 2026-09-24 15:27:00 on dist-test-slave-l766"
server_key: "1e5c9a5574058139d88cd02df3a1565a"
server_key_iv: "59a19f456179c81b83b57f4882555853"
server_key_version: "encryptionkey@0"
I20260924 15:27:00.839123 23815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:00.868932 23815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:00.871616 23815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:00.871803 23815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:00.872076 23815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:00.872437 23815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:00.872496 23815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:00.872538 23815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:00.872565 23815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:00.878669 23815 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:41417
I20260924 15:27:00.878715 23944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:41417 every 8 connection(s)
I20260924 15:27:00.879050 23815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:27:00.884506 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 23815
I20260924 15:27:00.884650 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:00.884866 4409 external_mini_cluster.cc:1428] Setting key 3476b07f5e2fab13f2a6fa07d98b7c70
Sep 24 15:27:00 dist-test-slave-l766 krb5kdc[23572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263620, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.66@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:00.889083 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:00.879756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:60027 (local address 127.4.78.126:40757)
0924 15:27:00.879811 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:00.879813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:00.880050 (+ 237us) server_negotiation.cc:407] Connection header received
0924 15:27:00.880538 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:00.880539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:00.880548 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:00.880611 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:00.881581 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.882151 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.883151 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.883336 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.883387 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:00.885885 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:00.885886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:00.885886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:00.885895 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:00.887593 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:00.887996 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:00.887998 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:00.887998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:00.888064 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:00.888295 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:00.888296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:00.888297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:00.888450 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:00.888521 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:00.888840 (+ 319us) server_negotiation.cc:299] Negotiation successful
0924 15:27:00.888941 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:00.889186 23947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:00.879731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40757 (local address 127.4.78.66:60027)
0924 15:27:00.879901 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:00.879945 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:00.880451 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:00.880687 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:00.880691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:00.881274 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:00.881498 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:00.881504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.882237 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.882238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:00.883071 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:00.883073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:00.883416 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:00.883416 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:00.883527 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:00.884013 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:00.884021 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:00.885761 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:00.887682 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:00.887689 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:00.887691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:00.887920 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:00.888133 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:00.888134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:00.888134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:00.888234 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:00.888555 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:00.888561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:00.888698 (+ 137us) client_negotiation.cc:769] Sending connection context
0924 15:27:00.888798 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 15:27:00.888954 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260924 15:27:00.889866 23945 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40757
I20260924 15:27:00.889991 23945 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:00.890180 23945 heartbeater.cc:511] Master 127.4.78.126:40757 requested a full tablet report, sending...
I20260924 15:27:00.890708 23617 ts_manager.cc:194] Registered new tserver with Master: 0a212890a18e41ea89370a12b311a5a0 (127.4.78.66:41417)
I20260924 15:27:00.891108 23617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.66@KRBTEST.COM'} at 127.4.78.66:60027
WARNING: no policy specified for kudu/127.4.78.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
Entry for principal kudu/127.4.78.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab.
I20260924 15:27:00.914327 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40757
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:00.995504 23950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:00.995855 23950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:00.995898 23950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:00.997159 23950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:00.997208 23950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:00.997236 23950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:27:00.998697 23950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.67.keytab
--principal=kudu/127.4.78.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:40757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:00.998934 23950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:00.999182 23950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:01.004171 23958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:01.004175 23955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:01.004338 23956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:01.004508 23950 server_base.cc:1034] running on GCE node
I20260924 15:27:01.004781 23950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:01.005039 23950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:01.006222 23950 hybrid_clock.cc:648] HybridClock initialized: now 1790263621006213 us; error 38 us; skew 500 ppm
Sep 24 15:27:01 dist-test-slave-l766 krb5kdc[23572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263621, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:01.009078 23950 init.cc:377] Logged in from keytab as kudu/127.4.78.67@KRBTEST.COM (short username kudu)
I20260924 15:27:01.009611 23950 webserver.cc:460] Webserver started at http://127.4.78.67:40083/ using document root <none> and password file <none>
I20260924 15:27:01.009907 23950 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:01.009979 23950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:01.010142 23950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:01.011232 23950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "895b2cf076aa490aad446a31184923d8"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "ce935177c79927e3d6093351e92c4516"
server_key_iv: "fe0fd2bc2893e58f6fb90cdef3d615c3"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.011612 23950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "895b2cf076aa490aad446a31184923d8"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "ce935177c79927e3d6093351e92c4516"
server_key_iv: "fe0fd2bc2893e58f6fb90cdef3d615c3"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.013002 23950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:01.013861 23965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.014056 23950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:01.014151 23950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "895b2cf076aa490aad446a31184923d8"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "ce935177c79927e3d6093351e92c4516"
server_key_iv: "fe0fd2bc2893e58f6fb90cdef3d615c3"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.014243 23950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:01.035472 23950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:01.038020 23950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:01.038187 23950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:01.038429 23950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:01.038757 23950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:01.038812 23950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.038856 23950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:01.038893 23950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.045053 23950 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:34553
I20260924 15:27:01.045094 24079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:34553 every 8 connection(s)
I20260924 15:27:01.045457 23950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:27:01.049497 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 23950
I20260924 15:27:01.049618 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:01.049775 4409 external_mini_cluster.cc:1428] Setting key e4b97b5dedb30dc9fc23197bc3066f3c
Sep 24 15:27:01 dist-test-slave-l766 krb5kdc[23572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263621, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.67@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:01.054709 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.046162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:51985 (local address 127.4.78.126:40757)
0924 15:27:01.046207 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.046209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.046412 (+ 203us) server_negotiation.cc:407] Connection header received
0924 15:27:01.046953 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.046954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.046968 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.047025 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:01.048000 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.048504 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.049554 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.049805 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.049922 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.052105 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:01.052106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:01.052107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:01.052113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:01.053539 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:01.053962 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:01.053963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:01.053963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:01.054015 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:01.054221 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:01.054221 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:01.054222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:01.054320 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:01.054368 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.054586 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.054646 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:01.054898 24082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.046123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:40757 (local address 127.4.78.67:51985)
0924 15:27:01.046281 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.046325 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:01.046835 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:01.047124 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:01.047133 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:01.047702 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:01.047898 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.047903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.048599 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.048600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.049462 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.049464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.049926 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.049927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.050113 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.050684 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:01.050693 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:01.052026 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:01.053616 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:01.053622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:01.053624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:01.053895 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:01.054073 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:01.054073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:01.054074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:01.054167 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:01.054375 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:01.054379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:01.054493 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 15:27:01.054574 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:27:01.054690 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260924 15:27:01.055361 24080 heartbeater.cc:347] Connected to a master server at 127.4.78.126:40757
I20260924 15:27:01.055462 24080 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:01.055635 24080 heartbeater.cc:511] Master 127.4.78.126:40757 requested a full tablet report, sending...
I20260924 15:27:01.056023 23617 ts_manager.cc:194] Registered new tserver with Master: 895b2cf076aa490aad446a31184923d8 (127.4.78.67:34553)
I20260924 15:27:01.056428 23617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.78.67@KRBTEST.COM'} at 127.4.78.67:51985
I20260924 15:27:01.061502 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:27:01.068645 23676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.062867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.4.78.126:40757)
0924 15:27:01.062950 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.062952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.063071 (+ 119us) server_negotiation.cc:407] Connection header received
0924 15:27:01.063146 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.063147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.063157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.063201 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:01.063730 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.064197 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.064927 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.065129 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.065201 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.066359 (+ 1158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:01.066360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:01.066360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:01.066368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:01.067484 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:01.067838 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:01.067839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:01.067840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:01.067898 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:01.068097 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:01.068098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:01.068098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:01.068243 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:01.068349 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.068478 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.068551 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:27:01.070345 23617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54222:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:01.071894 23617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:01.078608 24096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.074253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:40363 (local address 127.0.0.1:55564)
0924 15:27:01.074593 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.074605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:01.074672 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:01.075088 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:01.075091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:01.075096 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:01.075187 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.075189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.076100 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.076101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.077320 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.077322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.078357 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.078358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.078421 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.078426 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:27:01.078473 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:27:01.078517 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":2,"thread_start_us":121,"threads_started":1}
I20260924 15:27:01.078694 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.074588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.4.78.66:41417)
0924 15:27:01.074767 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.074772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.074825 (+ 53us) server_negotiation.cc:407] Connection header received
0924 15:27:01.074899 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.074901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.074917 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.074992 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:01.075256 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.076364 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.077483 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.078123 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.078211 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.078273 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.078563 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.078609 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":96,"threads_started":1}
I20260924 15:27:01.078891 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.074361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55564 (local address 127.4.78.65:40363)
0924 15:27:01.074821 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.074825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.074834 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:27:01.074864 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.074865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.074877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.074963 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:01.075283 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.075990 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.077409 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.078190 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.078262 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.078319 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.078729 (+ 410us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.078785 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":86,"threads_started":1}
I20260924 15:27:01.078678 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.074209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:34553 (local address 127.0.0.1:53340)
0924 15:27:01.074534 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.074575 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:01.074672 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:01.075088 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:01.075090 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:01.075092 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:01.075179 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.075181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.076472 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.076473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.077547 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.077549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.078425 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.078426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.078479 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.078482 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:01.078542 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:27:01.078578 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":53,"thread_start_us":47,"threads_started":1}
I20260924 15:27:01.079061 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.074367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.4.78.67:34553)
0924 15:27:01.074509 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.074514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.074651 (+ 137us) server_negotiation.cc:407] Connection header received
0924 15:27:01.074923 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.074925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.074942 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.075008 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:01.075254 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.076364 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.077624 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.078326 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.078397 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.078447 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.078557 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.078598 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260924 15:27:01.079181 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.074392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:41417 (local address 127.0.0.1:42276)
0924 15:27:01.074777 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.074785 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:01.074834 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:01.075087 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:01.075090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:01.075095 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:01.075168 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.075171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.076484 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.076485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.077393 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.077396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.078210 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.078211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.078301 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.078315 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:27:01.078416 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 15:27:01.078473 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":4,"thread_start_us":60,"threads_started":1}
I20260924 15:27:01.079780 23878 tablet_service.cc:1467] Processing CreateTablet for tablet f11aa70a20fc4c76b6c6ea2edb181b7d (DEFAULT_TABLE table=test-table [id=22166823599b45fba7407f3a19974215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:01.079753 24013 tablet_service.cc:1467] Processing CreateTablet for tablet f11aa70a20fc4c76b6c6ea2edb181b7d (DEFAULT_TABLE table=test-table [id=22166823599b45fba7407f3a19974215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:01.080116 24013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11aa70a20fc4c76b6c6ea2edb181b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:01.080116 23878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11aa70a20fc4c76b6c6ea2edb181b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:01.080678 23743 tablet_service.cc:1467] Processing CreateTablet for tablet f11aa70a20fc4c76b6c6ea2edb181b7d (DEFAULT_TABLE table=test-table [id=22166823599b45fba7407f3a19974215]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:01.080976 23743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f11aa70a20fc4c76b6c6ea2edb181b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:01.083469 24098 tablet_bootstrap.cc:492] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0: Bootstrap starting.
I20260924 15:27:01.083808 24100 tablet_bootstrap.cc:492] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18: Bootstrap starting.
I20260924 15:27:01.084127 24098 tablet_bootstrap.cc:654] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:01.084384 24100 tablet_bootstrap.cc:654] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:01.084690 24098 log.cc:824] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:01.084916 24100 log.cc:824] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:01.085027 24099 tablet_bootstrap.cc:492] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Bootstrap starting.
I20260924 15:27:01.085489 24098 tablet_bootstrap.cc:492] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0: No bootstrap required, opened a new log
I20260924 15:27:01.085565 24098 ts_tablet_manager.cc:1397] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:27:01.085672 24099 tablet_bootstrap.cc:654] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:01.086179 24099 log.cc:824] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:01.086833 24099 tablet_bootstrap.cc:492] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: No bootstrap required, opened a new log
I20260924 15:27:01.086913 24099 ts_tablet_manager.cc:1397] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:27:01.087109 24098 raft_consensus.cc:348] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.087244 24098 raft_consensus.cc:374] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:01.087273 24098 raft_consensus.cc:729] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a212890a18e41ea89370a12b311a5a0, State: Initialized, Role: FOLLOWER
I20260924 15:27:01.087407 24098 consensus_queue.cc:260] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.087709 24098 ts_tablet_manager.cc:1428] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:27:01.087744 23945 heartbeater.cc:503] Master 127.4.78.126:40757 was elected leader, sending a full tablet report...
I20260924 15:27:01.088114 24100 tablet_bootstrap.cc:492] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18: No bootstrap required, opened a new log
I20260924 15:27:01.088186 24100 ts_tablet_manager.cc:1397] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 15:27:01.088752 24099 raft_consensus.cc:348] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.088932 24099 raft_consensus.cc:374] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:01.088977 24099 raft_consensus.cc:729] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895b2cf076aa490aad446a31184923d8, State: Initialized, Role: FOLLOWER
I20260924 15:27:01.089093 24099 consensus_queue.cc:260] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.089401 24080 heartbeater.cc:503] Master 127.4.78.126:40757 was elected leader, sending a full tablet report...
I20260924 15:27:01.089385 24099 ts_tablet_manager.cc:1428] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:27:01.090365 24100 raft_consensus.cc:348] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.090528 24100 raft_consensus.cc:374] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:01.090575 24100 raft_consensus.cc:729] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d042aa6ae94bdfb856b4a7bab4ad18, State: Initialized, Role: FOLLOWER
I20260924 15:27:01.090679 24100 consensus_queue.cc:260] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.090935 24100 ts_tablet_manager.cc:1428] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:01.091032 23810 heartbeater.cc:503] Master 127.4.78.126:40757 was elected leader, sending a full tablet report...
W20260924 15:27:01.129673 23946 tablet.cc:2369] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:27:01.205106 23811 tablet.cc:2369] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:27:01.296029 24081 tablet.cc:2369] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:01.499969 24106 raft_consensus.cc:482] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:01.500180 24106 raft_consensus.cc:504] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.500542 24106 leader_election.cc:288] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 895b2cf076aa490aad446a31184923d8 (127.4.78.67:34553), 0a212890a18e41ea89370a12b311a5a0 (127.4.78.66:41417)
I20260924 15:27:01.503746 24094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.500856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:39137 (local address 127.4.78.67:34553)
0924 15:27:01.500951 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.500954 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.501055 (+ 101us) server_negotiation.cc:407] Connection header received
0924 15:27:01.501127 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.501128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.501141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.501262 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:01.501442 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.502032 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.503010 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.503397 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.503483 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.503493 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.503639 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.503675 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 15:27:01.503758 24097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.501054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:46777 (local address 127.4.78.66:41417)
0924 15:27:01.501105 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.501107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.501115 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:01.501183 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.501185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.501195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.501287 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:01.501460 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.502035 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.503012 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.503386 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.503472 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.503482 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.503656 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.503690 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:27:01.503772 24107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.500761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:34553 (local address 127.4.78.65:39137)
0924 15:27:01.500959 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.500971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:01.501058 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:01.501271 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:01.501272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:01.501274 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:01.501357 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.501359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.502112 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.502113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.502916 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.502919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.503486 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.503486 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.503563 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.503566 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:01.503633 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:27:01.503675 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":4,"thread_start_us":86,"threads_started":1}
I20260924 15:27:01.503793 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.500956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:41417 (local address 127.4.78.65:46777)
0924 15:27:01.501054 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.501065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:01.501124 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:01.501294 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:01.501294 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:01.501296 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:01.501385 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.501387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.502122 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.502124 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.502914 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.502916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.503472 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.503473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.503578 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.503581 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:01.503650 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:27:01.503690 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":39,"threads_started":1}
I20260924 15:27:01.504165 24033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11aa70a20fc4c76b6c6ea2edb181b7d" candidate_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895b2cf076aa490aad446a31184923d8" is_pre_election: true
I20260924 15:27:01.504370 24033 raft_consensus.cc:2454] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d042aa6ae94bdfb856b4a7bab4ad18 in term 0.
I20260924 15:27:01.504652 23698 leader_election.cc:302] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d042aa6ae94bdfb856b4a7bab4ad18, 895b2cf076aa490aad446a31184923d8; no voters:
I20260924 15:27:01.504787 24106 raft_consensus.cc:2781] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:01.504773 23899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11aa70a20fc4c76b6c6ea2edb181b7d" candidate_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a212890a18e41ea89370a12b311a5a0" is_pre_election: true
I20260924 15:27:01.504839 24106 raft_consensus.cc:482] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:01.504863 24106 raft_consensus.cc:3037] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:01.504921 23899 raft_consensus.cc:2454] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d042aa6ae94bdfb856b4a7bab4ad18 in term 0.
I20260924 15:27:01.505651 24106 raft_consensus.cc:504] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.505784 24106 leader_election.cc:288] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [CANDIDATE]: Term 1 election: Requested vote from peers 895b2cf076aa490aad446a31184923d8 (127.4.78.67:34553), 0a212890a18e41ea89370a12b311a5a0 (127.4.78.66:41417)
I20260924 15:27:01.506004 23899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11aa70a20fc4c76b6c6ea2edb181b7d" candidate_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a212890a18e41ea89370a12b311a5a0"
I20260924 15:27:01.506004 24033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f11aa70a20fc4c76b6c6ea2edb181b7d" candidate_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "895b2cf076aa490aad446a31184923d8"
I20260924 15:27:01.506093 23899 raft_consensus.cc:3037] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:01.506109 24033 raft_consensus.cc:3037] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:01.507033 23899 raft_consensus.cc:2454] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d042aa6ae94bdfb856b4a7bab4ad18 in term 1.
I20260924 15:27:01.507033 24033 raft_consensus.cc:2454] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d042aa6ae94bdfb856b4a7bab4ad18 in term 1.
I20260924 15:27:01.507216 23698 leader_election.cc:302] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d042aa6ae94bdfb856b4a7bab4ad18, 895b2cf076aa490aad446a31184923d8; no voters:
I20260924 15:27:01.507347 24106 raft_consensus.cc:2781] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:01.507517 24106 raft_consensus.cc:686] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 1 LEADER]: Becoming Leader. State: Replica: 13d042aa6ae94bdfb856b4a7bab4ad18, State: Running, Role: LEADER
I20260924 15:27:01.507627 24106 consensus_queue.cc:237] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } }
I20260924 15:27:01.508497 23616 catalog_manager.cc:5495] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d042aa6ae94bdfb856b4a7bab4ad18 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d042aa6ae94bdfb856b4a7bab4ad18" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 40363 } health_report { overall_health: HEALTHY } } }
I20260924 15:27:01.517562 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.515146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55576 (local address 127.4.78.65:40363)
0924 15:27:01.515208 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.515211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.515219 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:01.515302 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.515303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.515323 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.515410 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:01.515536 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.515997 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.516791 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.516997 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.517064 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.517244 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:01.517348 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:01.517442 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.517466 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.517487 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:27:01.519322 24033 raft_consensus.cc:1261] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 1 FOLLOWER]: Refusing update from remote peer 13d042aa6ae94bdfb856b4a7bab4ad18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:01.519398 23899 raft_consensus.cc:1261] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 1 FOLLOWER]: Refusing update from remote peer 13d042aa6ae94bdfb856b4a7bab4ad18: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:01.519753 24109 consensus_queue.cc:1035] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "895b2cf076aa490aad446a31184923d8" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 34553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:01.519896 24106 consensus_queue.cc:1035] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a212890a18e41ea89370a12b311a5a0" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 41417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:01.523097 24114 mvcc.cc:205] Tried to move back new op lower bound from 7332919793740308480 to 7332919793695698944. Current Snapshot: MvccSnapshot[applied={T|T < 7332919793740308480}]
I20260924 15:27:01.523499 24113 mvcc.cc:205] Tried to move back new op lower bound from 7332919793740308480 to 7332919793695698944. Current Snapshot: MvccSnapshot[applied={T|T < 7332919793740308480 or (T in {7332919793740308480})}]
I20260924 15:27:01.525745 23616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:01.525878 23616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54222:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:01.526643 23616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5d097ba87b34331832dfeabee212e0c: Sending DeleteTablet for 3 replicas of tablet f11aa70a20fc4c76b6c6ea2edb181b7d
I20260924 15:27:01.526937 24013 tablet_service.cc:1514] Processing DeleteTablet for tablet f11aa70a20fc4c76b6c6ea2edb181b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:01 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:53340
I20260924 15:27:01.526973 23878 tablet_service.cc:1514] Processing DeleteTablet for tablet f11aa70a20fc4c76b6c6ea2edb181b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:01 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:42276
I20260924 15:27:01.527030 23743 tablet_service.cc:1514] Processing DeleteTablet for tablet f11aa70a20fc4c76b6c6ea2edb181b7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:01 UTC) from {username='kudu', principal='kudu/127.4.78.126@KRBTEST.COM'} at 127.0.0.1:55564
I20260924 15:27:01.527235 24124 tablet_replica.cc:330] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18: stopping tablet replica
I20260924 15:27:01.527299 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 23680
I20260924 15:27:01.527328 24124 raft_consensus.cc:2229] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:27:01.527416 24124 raft_consensus.cc:2258] T f11aa70a20fc4c76b6c6ea2edb181b7d P 13d042aa6ae94bdfb856b4a7bab4ad18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:01.533278 24123 tablet_replica.cc:330] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0: stopping tablet replica
I20260924 15:27:01.533392 24123 raft_consensus.cc:2229] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:01.533514 24123 raft_consensus.cc:2258] T f11aa70a20fc4c76b6c6ea2edb181b7d P 0a212890a18e41ea89370a12b311a5a0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 15:27:01.533532 23601 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:40363 (error 108)
I20260924 15:27:01.533680 24122 tablet_replica.cc:330] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: stopping tablet replica
I20260924 15:27:01.533776 24122 raft_consensus.cc:2229] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:01.533882 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 23815
I20260924 15:27:01.533914 24122 raft_consensus.cc:2258] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:01.534730 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.534244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:40363 (local address 127.0.0.1:55586)
0924 15:27:01.534646 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.534691 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:40363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380}
W20260924 15:27:01.534876 23601 catalog_manager.cc:4584] TS 13d042aa6ae94bdfb856b4a7bab4ad18 (127.4.78.65:40363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f11aa70a20fc4c76b6c6ea2edb181b7d: Network error: Client connection negotiation failed: client connection to 127.4.78.65:40363: connect: Connection refused (error 111)
I20260924 15:27:01.535791 24122 ts_tablet_manager.cc:1905] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:01.537333 24122 ts_tablet_manager.cc:1918] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:01.537424 24122 log.cc:1196] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/f11aa70a20fc4c76b6c6ea2edb181b7d
I20260924 15:27:01.537639 24122 ts_tablet_manager.cc:1939] T f11aa70a20fc4c76b6c6ea2edb181b7d P 895b2cf076aa490aad446a31184923d8: Deleting consensus metadata
I20260924 15:27:01.538081 23603 catalog_manager.cc:4841] TS 895b2cf076aa490aad446a31184923d8 (127.4.78.67:34553): tablet f11aa70a20fc4c76b6c6ea2edb181b7d (table test-table [id=22166823599b45fba7407f3a19974215]) successfully deleted
I20260924 15:27:01.539096 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 23950
I20260924 15:27:01.539157 24093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.539047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:41417 (local address 127.0.0.1:42290)
0924 15:27:01.539093 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.539126 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.66:41417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 15:27:01.546545 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 23585
2026-09-24T15:27:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2310 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T15:27:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:01Z Disabled control of system clock
I20260924 15:27:01.574028 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:43533
--webserver_interface=127.4.78.126
--webserver_port=0
--builtin_ntp_servers=127.4.78.84:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:43533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:27:01.655753 24129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:01.655958 24129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:01.655979 24129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:01.657063 24129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:01.657096 24129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:01.657109 24129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:01.657121 24129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:01.657137 24129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:01.658558 24129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:43533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:43533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:01.658749 24129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:01.658978 24129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:01.663710 24137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:01.663727 24129 server_base.cc:1034] running on GCE node
W20260924 15:27:01.663790 24135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:01.663699 24134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:01.664075 24129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:01.664397 24129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:01.665570 24129 hybrid_clock.cc:648] HybridClock initialized: now 1790263621665563 us; error 53 us; skew 500 ppm
I20260924 15:27:01.666821 24129 webserver.cc:460] Webserver started at http://127.4.78.126:38883/ using document root <none> and password file <none>
I20260924 15:27:01.667079 24129 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:01.667150 24129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:01.667309 24129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:01.668345 24129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "6b25199f23c2483c8ae7723edfb54249"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "e74f895a17a1a04faee14d2c9df1faa5"
server_key_iv: "27a2b772c3b834fe23a47810ccc08273"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.668695 24129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "6b25199f23c2483c8ae7723edfb54249"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "e74f895a17a1a04faee14d2c9df1faa5"
server_key_iv: "27a2b772c3b834fe23a47810ccc08273"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.670064 24129 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:01.670787 24143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.670979 24129 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:01.671084 24129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "6b25199f23c2483c8ae7723edfb54249"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "e74f895a17a1a04faee14d2c9df1faa5"
server_key_iv: "27a2b772c3b834fe23a47810ccc08273"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.671159 24129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:01.687633 24129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:01.688000 24129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:01.688167 24129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:01.692461 24129 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:43533
I20260924 15:27:01.692520 24196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:43533 every 8 connection(s)
I20260924 15:27:01.692917 24129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:01.694049 24197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:01.696529 24197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249: Bootstrap starting.
I20260924 15:27:01.697291 24197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:01.697635 24197 log.cc:824] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:01.698309 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 24129
I20260924 15:27:01.698396 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:01.698467 24197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249: No bootstrap required, opened a new log
I20260924 15:27:01.698614 4409 external_mini_cluster.cc:1428] Setting key cd65a3703d8b8a6584cb6706b7dbd08f
I20260924 15:27:01.701502 24197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b25199f23c2483c8ae7723edfb54249" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43533 } }
I20260924 15:27:01.701678 24197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:01.701706 24197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b25199f23c2483c8ae7723edfb54249, State: Initialized, Role: FOLLOWER
I20260924 15:27:01.701846 24197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b25199f23c2483c8ae7723edfb54249" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43533 } }
I20260924 15:27:01.701924 24197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:01.701954 24197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:01.701998 24197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:01.702574 24197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b25199f23c2483c8ae7723edfb54249" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43533 } }
I20260924 15:27:01.702663 24197 leader_election.cc:302] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b25199f23c2483c8ae7723edfb54249; no voters:
I20260924 15:27:01.702873 24197 leader_election.cc:288] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:01.702908 24202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:01.703018 24202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [term 1 LEADER]: Becoming Leader. State: Replica: 6b25199f23c2483c8ae7723edfb54249, State: Running, Role: LEADER
I20260924 15:27:01.703096 24202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b25199f23c2483c8ae7723edfb54249" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43533 } }
I20260924 15:27:01.703140 24197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:01.703397 24202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b25199f23c2483c8ae7723edfb54249. Latest consensus state: current_term: 1 leader_uuid: "6b25199f23c2483c8ae7723edfb54249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b25199f23c2483c8ae7723edfb54249" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43533 } } }
I20260924 15:27:01.703477 24202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:01.703408 24204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b25199f23c2483c8ae7723edfb54249" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b25199f23c2483c8ae7723edfb54249" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 43533 } } }
I20260924 15:27:01.703639 24204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:01.703815 24206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:01.704299 24206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:01.705474 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.699151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51874 (local address 127.4.78.126:43533)
0924 15:27:01.699392 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.699402 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.699435 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:27:01.700003 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.700013 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.700093 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.700252 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:01.700452 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.701950 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.702810 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.703462 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.703930 (+ 468us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.704400 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:01.704404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:01.704404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:01.704634 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:01.704685 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:01.704703 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:01.704780 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.705131 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.705216 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":91,"threads_started":1}
I20260924 15:27:01.705847 24219 mvcc.cc:205] Tried to move back new op lower bound from 7332919794504699904 to 7332919794496262144. Current Snapshot: MvccSnapshot[applied={T|T < 7332919794504699904}]
I20260924 15:27:01.706171 24206 catalog_manager.cc:1347] Generated new cluster ID: 361b74a5b9a64440a9ebcbd10d787a23
I20260924 15:27:01.706226 24206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:01.724403 24206 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:01.725188 24206 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:01.728822 24206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249: Generated new TSK 0
I20260924 15:27:01.729015 24206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:01.732913 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.65:0
--local_ip_for_outbound_sockets=127.4.78.65
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:43533
--builtin_ntp_servers=127.4.78.84:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:27:01.817055 24221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:01.817299 24221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:01.817335 24221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:01.818576 24221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:01.818638 24221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:01.818681 24221 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.65
I20260924 15:27:01.820029 24221 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.78.65
--webserver_port=0
--tserver_master_addrs=127.4.78.126:43533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24221
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.65
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:01.820288 24221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:01.820582 24221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:01.826339 24227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:01.826349 24226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:01.826542 24221 server_base.cc:1034] running on GCE node
W20260924 15:27:01.826579 24229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:01.826802 24221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:01.827080 24221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:01.828272 24221 hybrid_clock.cc:648] HybridClock initialized: now 1790263621828251 us; error 36 us; skew 500 ppm
I20260924 15:27:01.829726 24221 webserver.cc:460] Webserver started at http://127.4.78.65:43883/ using document root <none> and password file <none>
I20260924 15:27:01.830037 24221 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:01.830122 24221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:01.830319 24221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:01.831501 24221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/data/instance:
uuid: "e3b8fc527dee46eaad151152ca4000b7"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "f744197e7eacb1c9e9feae6fc456f77e"
server_key_iv: "abc284fce901ac1dea9de16bdd84b812"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.831894 24221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance:
uuid: "e3b8fc527dee46eaad151152ca4000b7"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "f744197e7eacb1c9e9feae6fc456f77e"
server_key_iv: "abc284fce901ac1dea9de16bdd84b812"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.833362 24221 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:01.834319 24235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.834493 24221 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:27:01.834609 24221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/wal
uuid: "e3b8fc527dee46eaad151152ca4000b7"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "f744197e7eacb1c9e9feae6fc456f77e"
server_key_iv: "abc284fce901ac1dea9de16bdd84b812"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.834690 24221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:01.848320 24221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:01.848737 24221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:01.848907 24221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:01.849220 24221 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:01.849637 24221 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:01.849691 24221 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.849745 24221 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:01.849772 24221 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.856690 24221 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.65:34361
I20260924 15:27:01.856762 24350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.65:34361 every 8 connection(s)
I20260924 15:27:01.857117 24221 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/data/info.pb
I20260924 15:27:01.858459 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 24221
I20260924 15:27:01.858541 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-0/wal/instance
I20260924 15:27:01.858738 4409 external_mini_cluster.cc:1428] Setting key dd6e335454869be3c3d48445ee7cdd54
I20260924 15:27:01.859810 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.66:0
--local_ip_for_outbound_sockets=127.4.78.66
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:43533
--builtin_ntp_servers=127.4.78.84:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:27:01.864063 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.857936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:34659 (local address 127.4.78.126:43533)
0924 15:27:01.857999 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.858001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.858261 (+ 260us) server_negotiation.cc:407] Connection header received
0924 15:27:01.859013 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.859014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.859047 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.859122 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:01.860067 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.860669 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.862381 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.862727 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.862807 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.863723 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:01.863725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:01.863726 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:01.863731 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:01.863749 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:01.863754 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:01.863840 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.863915 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.863945 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:27:01.864184 24353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.857953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:43533 (local address 127.4.78.65:34659)
0924 15:27:01.858125 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.858162 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:01.858887 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:01.859197 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:01.859202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:01.859213 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:01.859936 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.859945 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.860809 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.860810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.862294 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.862297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.862829 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.862830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.862955 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.863576 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:01.863577 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:01.863608 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:01.863631 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:01.863841 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:01.863843 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:01.863845 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:27:01.863897 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:27:01.863976 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260924 15:27:01.864813 24351 heartbeater.cc:347] Connected to a master server at 127.4.78.126:43533
I20260924 15:27:01.864935 24351 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:01.865150 24351 heartbeater.cc:511] Master 127.4.78.126:43533 requested a full tablet report, sending...
I20260924 15:27:01.865727 24160 ts_manager.cc:194] Registered new tserver with Master: e3b8fc527dee46eaad151152ca4000b7 (127.4.78.65:34361)
I20260924 15:27:01.866745 24160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.65:34659
W20260924 15:27:01.943746 24354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:01.943960 24354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:01.943984 24354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:01.945323 24354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:01.945369 24354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:01.945394 24354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.66
I20260924 15:27:01.946717 24354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.78.66
--webserver_port=0
--tserver_master_addrs=127.4.78.126:43533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.66
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:01.946923 24354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:01.947166 24354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:01.952389 24362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:01.952384 24360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:01.952395 24359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:01.952770 24354 server_base.cc:1034] running on GCE node
I20260924 15:27:01.952929 24354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:01.953125 24354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:01.954272 24354 hybrid_clock.cc:648] HybridClock initialized: now 1790263621954259 us; error 28 us; skew 500 ppm
I20260924 15:27:01.955511 24354 webserver.cc:460] Webserver started at http://127.4.78.66:41389/ using document root <none> and password file <none>
I20260924 15:27:01.955798 24354 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:01.955848 24354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:01.956120 24354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:01.957264 24354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/data/instance:
uuid: "d48049faa0f044efb0fb411fddf999d1"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "60d36607fa442d82399fa8272de7cb1a"
server_key_iv: "52efd692c33ecaf82acb52b24d0719eb"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.957652 24354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance:
uuid: "d48049faa0f044efb0fb411fddf999d1"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "60d36607fa442d82399fa8272de7cb1a"
server_key_iv: "52efd692c33ecaf82acb52b24d0719eb"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.959116 24354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:01.959862 24368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.960028 24354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:01.960134 24354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/wal
uuid: "d48049faa0f044efb0fb411fddf999d1"
format_stamp: "Formatted at 2026-09-24 15:27:01 on dist-test-slave-l766"
server_key: "60d36607fa442d82399fa8272de7cb1a"
server_key_iv: "52efd692c33ecaf82acb52b24d0719eb"
server_key_version: "encryptionkey@0"
I20260924 15:27:01.960211 24354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:01.972345 24354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:01.972678 24354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:01.972838 24354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:01.973080 24354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:01.973407 24354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:01.973462 24354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.973505 24354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:01.973531 24354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:01.979995 24354 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.66:44745
I20260924 15:27:01.980048 24482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.66:44745 every 8 connection(s)
I20260924 15:27:01.980418 24354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/data/info.pb
I20260924 15:27:01.985129 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 24354
I20260924 15:27:01.985219 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/wal/instance
I20260924 15:27:01.985459 4409 external_mini_cluster.cc:1428] Setting key 4af94c2dd06e07a813b5820d07cde130
I20260924 15:27:01.986150 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.981261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.66:51777 (local address 127.4.78.126:43533)
0924 15:27:01.981350 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:01.981352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:01.981596 (+ 244us) server_negotiation.cc:407] Connection header received
0924 15:27:01.982171 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:01.982172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:01.982183 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:01.982262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:01.982938 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.983460 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.984468 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.984654 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.984709 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.985772 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:01.985773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:01.985773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:01.985778 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:01.985792 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:01.985795 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:01.985933 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:01.986040 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:27:01.986064 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 15:27:01.986413 24485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:01.981246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:43533 (local address 127.4.78.66:51777)
0924 15:27:01.981462 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:01.981497 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:01.982068 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:01.982338 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:01.982341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:01.982349 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:01.982852 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.982857 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.983565 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.983566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.984371 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:01.984374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:01.984737 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:01.984738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:01.984851 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:01.985506 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:01.985507 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:01.985546 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:01.985573 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:01.985954 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:01.985960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:01.985963 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:01.986030 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:27:01.986155 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":117,"threads_started":1}
I20260924 15:27:01.986888 24483 heartbeater.cc:347] Connected to a master server at 127.4.78.126:43533
I20260924 15:27:01.986931 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.78.67:0
--local_ip_for_outbound_sockets=127.4.78.67
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:43533
--builtin_ntp_servers=127.4.78.84:43945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:27:01.986985 24483 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:01.987187 24483 heartbeater.cc:511] Master 127.4.78.126:43533 requested a full tablet report, sending...
I20260924 15:27:01.987586 24160 ts_manager.cc:194] Registered new tserver with Master: d48049faa0f044efb0fb411fddf999d1 (127.4.78.66:44745)
I20260924 15:27:01.987977 24160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.66:51777
W20260924 15:27:02.068843 24486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:02.069068 24486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:02.069105 24486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:02.070339 24486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:02.070398 24486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:02.070442 24486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.78.67
I20260924 15:27:02.071789 24486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.78.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.78.67
--webserver_port=0
--tserver_master_addrs=127.4.78.126:43533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.78.67
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:02.072049 24486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:02.072398 24486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:02.077400 24492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:02.077553 24486 server_base.cc:1034] running on GCE node
W20260924 15:27:02.077425 24494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:02.077420 24491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:02.077888 24486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:02.078188 24486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:02.079361 24486 hybrid_clock.cc:648] HybridClock initialized: now 1790263622079349 us; error 25 us; skew 500 ppm
I20260924 15:27:02.080631 24486 webserver.cc:460] Webserver started at http://127.4.78.67:46275/ using document root <none> and password file <none>
I20260924 15:27:02.080888 24486 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:02.080961 24486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:02.081102 24486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:02.082310 24486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/data/instance:
uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab"
format_stamp: "Formatted at 2026-09-24 15:27:02 on dist-test-slave-l766"
server_key: "256dfd07cf648d519b66db207844cb07"
server_key_iv: "1206ff1e940ba76c6cc22aa4b91c4cbe"
server_key_version: "encryptionkey@0"
I20260924 15:27:02.082671 24486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance:
uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab"
format_stamp: "Formatted at 2026-09-24 15:27:02 on dist-test-slave-l766"
server_key: "256dfd07cf648d519b66db207844cb07"
server_key_iv: "1206ff1e940ba76c6cc22aa4b91c4cbe"
server_key_version: "encryptionkey@0"
I20260924 15:27:02.083994 24486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:02.084808 24500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:02.085000 24486 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:02.085100 24486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/wal
uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab"
format_stamp: "Formatted at 2026-09-24 15:27:02 on dist-test-slave-l766"
server_key: "256dfd07cf648d519b66db207844cb07"
server_key_iv: "1206ff1e940ba76c6cc22aa4b91c4cbe"
server_key_version: "encryptionkey@0"
I20260924 15:27:02.085183 24486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:02.095108 24486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:02.095423 24486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:02.095579 24486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:02.095808 24486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:27:02.096129 24486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:27:02.096184 24486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:02.096233 24486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:27:02.096262 24486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:02.101949 24486 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.67:41077
I20260924 15:27:02.102027 24614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.67:41077 every 8 connection(s)
I20260924 15:27:02.102372 24486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/data/info.pb
I20260924 15:27:02.107925 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.103191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.67:34401 (local address 127.4.78.126:43533)
0924 15:27:02.103261 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:02.103263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:02.103532 (+ 269us) server_negotiation.cc:407] Connection header received
0924 15:27:02.104143 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:02.104144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:02.104154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:02.104205 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:02.104895 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.105488 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.106607 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.106815 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.106876 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.107691 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:02.107691 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:02.107692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:02.107695 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:02.107708 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:02.107711 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:02.107756 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:02.107843 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 15:27:02.107865 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:27:02.108114 24617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.103183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:43533 (local address 127.4.78.67:34401)
0924 15:27:02.103409 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:02.103444 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:02.104051 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:02.104264 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:02.104267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:02.104280 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:02.104791 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.104795 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.105590 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.105591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.106509 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.106512 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.106902 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.106903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.107011 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.107526 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:27:02.107526 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:02.107554 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:27:02.107603 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:02.107771 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:02.107775 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:02.107778 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:02.107833 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 15:27:02.107912 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":85,"threads_started":1}
I20260924 15:27:02.108661 24615 heartbeater.cc:347] Connected to a master server at 127.4.78.126:43533
I20260924 15:27:02.108770 24615 heartbeater.cc:464] Registering TS with master...
I20260924 15:27:02.108961 24615 heartbeater.cc:511] Master 127.4.78.126:43533 requested a full tablet report, sending...
I20260924 15:27:02.109429 24160 ts_manager.cc:194] Registered new tserver with Master: bbf73cee7a9847e9b1fec1f8ad0e9fab (127.4.78.67:41077)
I20260924 15:27:02.109879 24160 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.78.67:34401
I20260924 15:27:02.112131 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 24486
I20260924 15:27:02.112214 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/wal/instance
I20260924 15:27:02.112353 4409 external_mini_cluster.cc:1428] Setting key 0f47d72de54ea77bb14cf10a526ee12d
I20260924 15:27:02.113337 4409 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:27:02.117700 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.114838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51888 (local address 127.4.78.126:43533)
0924 15:27:02.114884 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:02.114886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:02.115040 (+ 154us) server_negotiation.cc:407] Connection header received
0924 15:27:02.115060 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:02.115060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:02.115069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:02.115111 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:02.115338 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.115807 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.116536 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.116716 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.116775 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.117442 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:02.117444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:02.117444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:02.117448 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:02.117461 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:02.117465 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:02.117522 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:02.117571 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 15:27:02.117592 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:02.119360 24160 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51888:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:27:02.120980 24160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:27:02.127287 24628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.123204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:41077 (local address 127.0.0.1:42210)
0924 15:27:02.123678 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:02.123689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:02.123743 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:02.124238 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:02.124238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:02.124240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:02.124289 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.124290 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.125179 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.125180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.126227 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.126230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.126983 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.126984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.127065 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.127082 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 15:27:02.127157 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:27:02.127205 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":1,"thread_start_us":426,"threads_started":1}
I20260924 15:27:02.127306 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.123431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42210 (local address 127.4.78.67:41077)
0924 15:27:02.124068 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:02.124073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:02.124082 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:27:02.124122 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:02.124124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:02.124139 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:02.124365 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:02.124411 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.125090 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.126317 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.126895 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.126969 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.127065 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:02.127180 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 15:27:02.127221 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":88,"threads_started":1}
I20260924 15:27:02.128029 24627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.123205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:44745 (local address 127.0.0.1:45468)
0924 15:27:02.123825 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:02.123833 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:02.123884 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:02.124143 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:02.124144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:02.124145 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:02.124178 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.124179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.125477 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.125478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.126689 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.126691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.127790 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.127791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.127837 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.127840 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:27:02.127913 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:27:02.127956 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":52,"spinlock_wait_cycles":768,"thread_start_us":431,"threads_started":1}
I20260924 15:27:02.128044 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.123385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45468 (local address 127.4.78.66:44745)
0924 15:27:02.123578 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:02.123583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:02.123907 (+ 324us) server_negotiation.cc:407] Connection header received
0924 15:27:02.123970 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:02.123973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:02.123990 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:02.124069 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:02.124260 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.125368 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.126817 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.127601 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.127668 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.127761 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:02.127928 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 15:27:02.127968 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":120,"threads_started":1}
I20260924 15:27:02.128120 24548 tablet_service.cc:1467] Processing CreateTablet for tablet acaab20ea480494aa4d342eaf3c964eb (DEFAULT_TABLE table=test-table [id=d97ea54bb8264d1ca455fcfe453a958d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:02.128548 24548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaab20ea480494aa4d342eaf3c964eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:02.128628 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.123198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:34361 (local address 127.0.0.1:49948)
0924 15:27:02.123501 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:02.123544 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:02.123615 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:02.123959 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:02.123982 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:02.123985 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:02.124061 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.124063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.125483 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.125484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.126698 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.126700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.128383 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.128385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.128448 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.128453 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:27:02.128509 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:27:02.128553 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":41,"thread_start_us":76,"threads_started":1}
I20260924 15:27:02.128921 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.123310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49948 (local address 127.4.78.65:34361)
0924 15:27:02.123771 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:02.123775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:02.123788 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:27:02.123836 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:02.123838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:02.123857 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:02.123942 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:02.124152 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.125397 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.126787 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.127894 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.127967 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.128049 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:02.128791 (+ 742us) server_negotiation.cc:299] Negotiation successful
0924 15:27:02.128840 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":83,"threads_started":1}
I20260924 15:27:02.129567 24284 tablet_service.cc:1467] Processing CreateTablet for tablet acaab20ea480494aa4d342eaf3c964eb (DEFAULT_TABLE table=test-table [id=d97ea54bb8264d1ca455fcfe453a958d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:02.129942 24284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaab20ea480494aa4d342eaf3c964eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:02.131481 24629 tablet_bootstrap.cc:492] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Bootstrap starting.
I20260924 15:27:02.132133 24629 tablet_bootstrap.cc:654] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:02.132150 24416 tablet_service.cc:1467] Processing CreateTablet for tablet acaab20ea480494aa4d342eaf3c964eb (DEFAULT_TABLE table=test-table [id=d97ea54bb8264d1ca455fcfe453a958d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:27:02.132506 24629 log.cc:824] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:02.132507 24416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acaab20ea480494aa4d342eaf3c964eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:02.132923 24631 tablet_bootstrap.cc:492] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7: Bootstrap starting.
I20260924 15:27:02.133278 24629 tablet_bootstrap.cc:492] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: No bootstrap required, opened a new log
I20260924 15:27:02.133339 24629 ts_tablet_manager.cc:1397] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:02.133575 24631 tablet_bootstrap.cc:654] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:02.133919 24631 log.cc:824] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:02.134893 24631 tablet_bootstrap.cc:492] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7: No bootstrap required, opened a new log
I20260924 15:27:02.134963 24631 ts_tablet_manager.cc:1397] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:27:02.135613 24633 tablet_bootstrap.cc:492] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Bootstrap starting.
I20260924 15:27:02.136129 24633 tablet_bootstrap.cc:654] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:02.136101 24629 raft_consensus.cc:348] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.136276 24629 raft_consensus.cc:374] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:02.136322 24629 raft_consensus.cc:729] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbf73cee7a9847e9b1fec1f8ad0e9fab, State: Initialized, Role: FOLLOWER
I20260924 15:27:02.136437 24629 consensus_queue.cc:260] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.136458 24633 log.cc:824] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:02.136689 24629 ts_tablet_manager.cc:1428] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260924 15:27:02.136829 24615 heartbeater.cc:503] Master 127.4.78.126:43533 was elected leader, sending a full tablet report...
I20260924 15:27:02.137380 24633 tablet_bootstrap.cc:492] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: No bootstrap required, opened a new log
I20260924 15:27:02.137449 24633 ts_tablet_manager.cc:1397] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:27:02.137531 24631 raft_consensus.cc:348] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.137692 24631 raft_consensus.cc:374] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:02.137737 24631 raft_consensus.cc:729] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3b8fc527dee46eaad151152ca4000b7, State: Initialized, Role: FOLLOWER
I20260924 15:27:02.137900 24631 consensus_queue.cc:260] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.138190 24631 ts_tablet_manager.cc:1428] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:27:02.138229 24351 heartbeater.cc:503] Master 127.4.78.126:43533 was elected leader, sending a full tablet report...
I20260924 15:27:02.139806 24633 raft_consensus.cc:348] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.139926 24633 raft_consensus.cc:374] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:02.139947 24633 raft_consensus.cc:729] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d48049faa0f044efb0fb411fddf999d1, State: Initialized, Role: FOLLOWER
I20260924 15:27:02.140053 24633 consensus_queue.cc:260] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.140280 24633 ts_tablet_manager.cc:1428] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:27:02.140395 24483 heartbeater.cc:503] Master 127.4.78.126:43533 was elected leader, sending a full tablet report...
W20260924 15:27:02.231017 24484 tablet.cc:2369] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:27:02.353047 24616 tablet.cc:2369] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:27:02.357951 24352 tablet.cc:2369] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:27:02.366740 24636 raft_consensus.cc:482] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:27:02.366853 24636 raft_consensus.cc:504] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.367238 24636 leader_election.cc:288] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bbf73cee7a9847e9b1fec1f8ad0e9fab (127.4.78.67:41077), d48049faa0f044efb0fb411fddf999d1 (127.4.78.66:44745)
I20260924 15:27:02.370391 24626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.367527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:47661 (local address 127.4.78.67:41077)
0924 15:27:02.367579 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:02.367581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:02.367629 (+ 48us) server_negotiation.cc:407] Connection header received
0924 15:27:02.367734 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:02.367735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:02.367746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:02.367824 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:02.368004 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.368507 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.369572 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.369998 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.370054 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.370062 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:02.370257 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 15:27:02.370313 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:27:02.370407 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.367510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.78.65:59767 (local address 127.4.78.66:44745)
0924 15:27:02.367596 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:02.367598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:02.367628 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:27:02.367741 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:02.367742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:02.367756 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:02.367833 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:27:02.368040 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.368507 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.369582 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.370066 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.370135 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.370145 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:02.370299 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:27:02.370340 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 15:27:02.370419 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.367421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.67:41077 (local address 127.4.78.65:47661)
0924 15:27:02.367559 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:02.367572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:02.367643 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:02.367831 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:02.367833 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:02.367835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:02.367929 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.367931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.368600 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.368602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.369472 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.369474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.370078 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.370080 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.370170 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.370174 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:02.370228 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:27:02.370289 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":30,"thread_start_us":78,"threads_started":1}
I20260924 15:27:02.370429 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.367407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.66:44745 (local address 127.4.78.65:59767)
0924 15:27:02.367558 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:02.367571 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:02.367641 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:02.367843 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:02.367844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:02.367846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:27:02.367939 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.367941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.368601 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.368602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.369473 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:02.369475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.370145 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.370146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:02.370199 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.370203 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:27:02.370285 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:27:02.370323 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":44,"thread_start_us":65,"threads_started":1}
I20260924 15:27:02.370795 24568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaab20ea480494aa4d342eaf3c964eb" candidate_uuid: "e3b8fc527dee46eaad151152ca4000b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" is_pre_election: true
I20260924 15:27:02.370795 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaab20ea480494aa4d342eaf3c964eb" candidate_uuid: "e3b8fc527dee46eaad151152ca4000b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48049faa0f044efb0fb411fddf999d1" is_pre_election: true
I20260924 15:27:02.370958 24568 raft_consensus.cc:2454] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3b8fc527dee46eaad151152ca4000b7 in term 0.
I20260924 15:27:02.370958 24436 raft_consensus.cc:2454] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3b8fc527dee46eaad151152ca4000b7 in term 0.
I20260924 15:27:02.371222 24238 leader_election.cc:302] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf73cee7a9847e9b1fec1f8ad0e9fab, e3b8fc527dee46eaad151152ca4000b7; no voters:
I20260924 15:27:02.371400 24636 raft_consensus.cc:2781] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:27:02.371479 24636 raft_consensus.cc:482] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:27:02.371515 24636 raft_consensus.cc:3037] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:02.372299 24636 raft_consensus.cc:504] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.372465 24636 leader_election.cc:288] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [CANDIDATE]: Term 1 election: Requested vote from peers bbf73cee7a9847e9b1fec1f8ad0e9fab (127.4.78.67:41077), d48049faa0f044efb0fb411fddf999d1 (127.4.78.66:44745)
I20260924 15:27:02.372627 24436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaab20ea480494aa4d342eaf3c964eb" candidate_uuid: "e3b8fc527dee46eaad151152ca4000b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d48049faa0f044efb0fb411fddf999d1"
I20260924 15:27:02.372725 24568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acaab20ea480494aa4d342eaf3c964eb" candidate_uuid: "e3b8fc527dee46eaad151152ca4000b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab"
I20260924 15:27:02.372723 24436 raft_consensus.cc:3037] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:02.372818 24568 raft_consensus.cc:3037] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:02.373538 24436 raft_consensus.cc:2454] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3b8fc527dee46eaad151152ca4000b7 in term 1.
I20260924 15:27:02.373554 24568 raft_consensus.cc:2454] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3b8fc527dee46eaad151152ca4000b7 in term 1.
I20260924 15:27:02.373699 24238 leader_election.cc:302] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbf73cee7a9847e9b1fec1f8ad0e9fab, e3b8fc527dee46eaad151152ca4000b7; no voters:
I20260924 15:27:02.373791 24636 raft_consensus.cc:2781] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:02.373955 24636 raft_consensus.cc:686] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [term 1 LEADER]: Becoming Leader. State: Replica: e3b8fc527dee46eaad151152ca4000b7, State: Running, Role: LEADER
I20260924 15:27:02.374054 24636 consensus_queue.cc:237] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } }
I20260924 15:27:02.374914 24159 catalog_manager.cc:5495] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3b8fc527dee46eaad151152ca4000b7 (127.4.78.65). New cstate: current_term: 1 leader_uuid: "e3b8fc527dee46eaad151152ca4000b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3b8fc527dee46eaad151152ca4000b7" member_type: VOTER last_known_addr { host: "127.4.78.65" port: 34361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 } health_report { overall_health: UNKNOWN } } }
I20260924 15:27:02.410420 24624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.408150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.4.78.65:34361)
0924 15:27:02.408216 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:02.408218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:02.408226 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:02.408266 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:02.408267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:02.408288 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:02.408351 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:02.408456 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.408927 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.409672 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:02.409892 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:02.409949 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:02.410086 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:02.410187 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:02.410248 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:02.410281 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 15:27:02.410299 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:27:02.412740 24568 raft_consensus.cc:1261] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 1 FOLLOWER]: Refusing update from remote peer e3b8fc527dee46eaad151152ca4000b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:02.412748 24436 raft_consensus.cc:1261] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 1 FOLLOWER]: Refusing update from remote peer e3b8fc527dee46eaad151152ca4000b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:02.413192 24640 consensus_queue.cc:1035] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbf73cee7a9847e9b1fec1f8ad0e9fab" member_type: VOTER last_known_addr { host: "127.4.78.67" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:02.413378 24636 consensus_queue.cc:1035] T acaab20ea480494aa4d342eaf3c964eb P e3b8fc527dee46eaad151152ca4000b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d48049faa0f044efb0fb411fddf999d1" member_type: VOTER last_known_addr { host: "127.4.78.66" port: 44745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:02.416177 24644 mvcc.cc:205] Tried to move back new op lower bound from 7332919797399023616 to 7332919797244678144. Current Snapshot: MvccSnapshot[applied={T|T < 7332919797399023616}]
I20260924 15:27:02.416445 24645 mvcc.cc:205] Tried to move back new op lower bound from 7332919797399023616 to 7332919797244678144. Current Snapshot: MvccSnapshot[applied={T|T < 7332919797399023616}]
I20260924 15:27:02.418674 24159 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:02.418782 24159 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51888:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:27:02.419647 24159 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b25199f23c2483c8ae7723edfb54249: Sending DeleteTablet for 3 replicas of tablet acaab20ea480494aa4d342eaf3c964eb
I20260924 15:27:02.420047 24284 tablet_service.cc:1514] Processing DeleteTablet for tablet acaab20ea480494aa4d342eaf3c964eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:02 UTC) from {username='slave'} at 127.0.0.1:49948
I20260924 15:27:02.420074 24416 tablet_service.cc:1514] Processing DeleteTablet for tablet acaab20ea480494aa4d342eaf3c964eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:02 UTC) from {username='slave'} at 127.0.0.1:45468
I20260924 15:27:02.420073 24548 tablet_service.cc:1514] Processing DeleteTablet for tablet acaab20ea480494aa4d342eaf3c964eb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:27:02 UTC) from {username='slave'} at 127.0.0.1:42210
I20260924 15:27:02.420279 24653 tablet_replica.cc:330] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: stopping tablet replica
I20260924 15:27:02.420377 24653 raft_consensus.cc:2229] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:02.420385 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 24221
I20260924 15:27:02.420485 24653 raft_consensus.cc:2258] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:02.421047 24653 ts_tablet_manager.cc:1905] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:02.421063 24655 tablet_replica.cc:330] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: stopping tablet replica
I20260924 15:27:02.421155 24655 raft_consensus.cc:2229] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:27:02.421283 24655 raft_consensus.cc:2258] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:27:02.421926 24655 ts_tablet_manager.cc:1905] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:27:02.423103 24653 ts_tablet_manager.cc:1918] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:02.423182 24653 log.cc:1196] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-1/wal/wals/acaab20ea480494aa4d342eaf3c964eb
I20260924 15:27:02.423445 24653 ts_tablet_manager.cc:1939] T acaab20ea480494aa4d342eaf3c964eb P d48049faa0f044efb0fb411fddf999d1: Deleting consensus metadata
I20260924 15:27:02.423971 24144 catalog_manager.cc:4841] TS d48049faa0f044efb0fb411fddf999d1 (127.4.78.66:44745): tablet acaab20ea480494aa4d342eaf3c964eb (table test-table [id=d97ea54bb8264d1ca455fcfe453a958d]) successfully deleted
I20260924 15:27:02.424117 24655 ts_tablet_manager.cc:1918] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:27:02.424191 24655 log.cc:1196] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Deleting WAL directory at /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263382361574-4409-0/minicluster-data/ts-2/wal/wals/acaab20ea480494aa4d342eaf3c964eb
I20260924 15:27:02.424456 24655 ts_tablet_manager.cc:1939] T acaab20ea480494aa4d342eaf3c964eb P bbf73cee7a9847e9b1fec1f8ad0e9fab: Deleting consensus metadata
I20260924 15:27:02.425083 24146 catalog_manager.cc:4841] TS bbf73cee7a9847e9b1fec1f8ad0e9fab (127.4.78.67:41077): tablet acaab20ea480494aa4d342eaf3c964eb (table test-table [id=d97ea54bb8264d1ca455fcfe453a958d]) successfully deleted
W20260924 15:27:02.426790 24147 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.78.65:34361 (error 108)
I20260924 15:27:02.427285 24623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:02.427153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.65:34361 (local address 127.0.0.1:42836)
0924 15:27:02.427208 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:02.427248 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.65:34361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:27:02.427332 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 24354
W20260924 15:27:02.427433 24147 catalog_manager.cc:4584] TS e3b8fc527dee46eaad151152ca4000b7 (127.4.78.65:34361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet acaab20ea480494aa4d342eaf3c964eb: Network error: Client connection negotiation failed: client connection to 127.4.78.65:34361: connect: Connection refused (error 111)
I20260924 15:27:02.432484 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 24486
I20260924 15:27:02.437234 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 24129
2026-09-24T15:27:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (890 ms)
[----------] 6 tests from EncryptionPolicyTest (8442 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:02 dist-test-slave-l766 krb5kdc[24659](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:02 dist-test-slave-l766 krb5kdc[24659](info): set up 2 sockets
Sep 24 15:27:02 dist-test-slave-l766 krb5kdc[24659](info): commencing operation
krb5kdc: starting...
W20260924 15:27:03.480710 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:03 dist-test-slave-l766 krb5kdc[24659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:03.600016 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:03.683706 24672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:03.683928 24672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:03.683952 24672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:03.685169 24672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:03.685216 24672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:03.685230 24672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:03.685276 24672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:03.685305 24672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:03.686858 24672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:03.687122 24672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:03.687378 24672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:03.692339 24677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:03.692407 24672 server_base.cc:1034] running on GCE node
W20260924 15:27:03.692329 24678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:03.692329 24681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:03.692782 24672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:03.693120 24672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:03.694286 24672 hybrid_clock.cc:648] HybridClock initialized: now 1790263623694260 us; error 36 us; skew 500 ppm
Sep 24 15:27:03 dist-test-slave-l766 krb5kdc[24659](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:03.696898 24672 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:03.697345 24672 webserver.cc:460] Webserver started at http://127.0.0.1:37447/ using document root <none> and password file <none>
I20260924 15:27:03.697599 24672 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:03.697672 24672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:03.697886 24672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:03.698959 24672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "17b2bfa8225248ed88d207d4d688844e"
format_stamp: "Formatted at 2026-09-24 15:27:03 on dist-test-slave-l766"
server_key: "7cef935e2d00cbe20de94e6c86c493d7"
server_key_iv: "1146d4313d28fa898faabb25fef4ede7"
server_key_version: "encryptionkey@0"
I20260924 15:27:03.699323 24672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "17b2bfa8225248ed88d207d4d688844e"
format_stamp: "Formatted at 2026-09-24 15:27:03 on dist-test-slave-l766"
server_key: "7cef935e2d00cbe20de94e6c86c493d7"
server_key_iv: "1146d4313d28fa898faabb25fef4ede7"
server_key_version: "encryptionkey@0"
I20260924 15:27:03.700651 24672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:03.701387 24688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:03.701570 24672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:03.701670 24672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "17b2bfa8225248ed88d207d4d688844e"
format_stamp: "Formatted at 2026-09-24 15:27:03 on dist-test-slave-l766"
server_key: "7cef935e2d00cbe20de94e6c86c493d7"
server_key_iv: "1146d4313d28fa898faabb25fef4ede7"
server_key_version: "encryptionkey@0"
I20260924 15:27:03.701756 24672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:03.714123 24672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:03.715957 24672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:03.716123 24672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:03.720069 24672 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37977
I20260924 15:27:03.720106 24740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37977 every 8 connection(s)
I20260924 15:27:03.720470 24672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:03.721455 24741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:03.723747 24741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e: Bootstrap starting.
I20260924 15:27:03.724330 24741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:03.724628 24741 log.cc:824] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:03.724720 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 24672
I20260924 15:27:03.724866 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:03.725081 4409 external_mini_cluster.cc:1428] Setting key 56c5b974072ae1c827c36446aceeb9fd
I20260924 15:27:03.725440 24741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e: No bootstrap required, opened a new log
I20260924 15:27:03.727555 24741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b2bfa8225248ed88d207d4d688844e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } }
I20260924 15:27:03.727766 24741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:03.727803 24741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17b2bfa8225248ed88d207d4d688844e, State: Initialized, Role: FOLLOWER
I20260924 15:27:03.727945 24741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b2bfa8225248ed88d207d4d688844e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } }
I20260924 15:27:03.728037 24741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:03.728068 24741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:03.728113 24741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:03.728813 24741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b2bfa8225248ed88d207d4d688844e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } }
I20260924 15:27:03.728920 24741 leader_election.cc:302] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17b2bfa8225248ed88d207d4d688844e; no voters:
I20260924 15:27:03.729069 24741 leader_election.cc:288] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:03.729130 24746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:03.729317 24746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [term 1 LEADER]: Becoming Leader. State: Replica: 17b2bfa8225248ed88d207d4d688844e, State: Running, Role: LEADER
I20260924 15:27:03.729346 24741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:03.729415 24746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b2bfa8225248ed88d207d4d688844e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } }
I20260924 15:27:03.729789 24746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17b2bfa8225248ed88d207d4d688844e. Latest consensus state: current_term: 1 leader_uuid: "17b2bfa8225248ed88d207d4d688844e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b2bfa8225248ed88d207d4d688844e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } } }
I20260924 15:27:03.729907 24746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:03.729797 24747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17b2bfa8225248ed88d207d4d688844e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17b2bfa8225248ed88d207d4d688844e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37977 } } }
I20260924 15:27:03.729993 24747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:03.730549 24753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:03.731065 24753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 15:27:03 dist-test-slave-l766 krb5kdc[24659](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:27:03.732610 24753 catalog_manager.cc:1347] Generated new cluster ID: 7b3772b31cba410d95bc14e3387db0bf
I20260924 15:27:03.733875 24753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:03.738112 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:03.725668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57265 (local address 127.0.0.1:37977)
0924 15:27:03.725899 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:03.725909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:03.725943 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:27:03.726778 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:03.726788 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:03.726853 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:03.727006 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:03.727631 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:03.729341 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:03.730462 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:03.731141 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:03.731260 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:03.734321 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:03.734325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:03.734325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:03.734337 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:03.736327 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:03.736799 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:03.736802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:03.736802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:03.736903 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:03.737180 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:03.737181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:03.737182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:03.737472 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:03.737628 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:03.737798 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 15:27:03.737920 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260924 15:27:03.739355 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:03.748241 24745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:03.740804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42973 (local address 127.0.0.1:37977)
0924 15:27:03.740837 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:03.740838 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:03.740951 (+ 113us) server_negotiation.cc:407] Connection header received
0924 15:27:03.741002 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:03.741003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:03.741012 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:03.741067 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:03.741720 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:03.742475 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:03.743513 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:03.743823 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:03.743884 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:03.745327 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:03.745329 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:03.745329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:03.745336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:03.746930 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:03.747381 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:03.747382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:03.747383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:03.747442 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:03.747688 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:03.747689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:03.747690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:03.747803 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:03.747924 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:03.748072 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 15:27:03.748136 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:27:03.748582 24753 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:03.749140 24753 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:03.751327 24753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17b2bfa8225248ed88d207d4d688844e: Generated new TSK 0
I20260924 15:27:03.751520 24753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:03.754047 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 24672
W20260924 15:27:03.758764 24657 connection.cc:535] client connection to 127.0.0.1:37977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37977): Connection reset by peer (error 104)
2026-09-24T15:27:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:03 dist-test-slave-l766 krb5kdc[24772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:03 dist-test-slave-l766 krb5kdc[24772](info): set up 2 sockets
Sep 24 15:27:03 dist-test-slave-l766 krb5kdc[24772](info): commencing operation
krb5kdc: starting...
W20260924 15:27:04.784905 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:04 dist-test-slave-l766 krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:04.905588 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:04.985723 24785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:04.985934 24785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:04.985957 24785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:04.987040 24785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:04.987076 24785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:04.987090 24785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:04.987103 24785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:04.987115 24785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:04.988498 24785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35311
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:04.988682 24785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:04.988843 24785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:04.993564 24791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:04.993573 24790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:04.993568 24793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:04.993784 24785 server_base.cc:1034] running on GCE node
I20260924 15:27:04.994035 24785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:04.994274 24785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:04.995448 24785 hybrid_clock.cc:648] HybridClock initialized: now 1790263624995435 us; error 43 us; skew 500 ppm
Sep 24 15:27:04 dist-test-slave-l766 krb5kdc[24772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263624, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:04.997938 24785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:04.998433 24785 webserver.cc:460] Webserver started at http://127.0.0.1:37949/ using document root <none> and password file <none>
I20260924 15:27:04.998678 24785 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:04.998752 24785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:04.998912 24785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:04.999935 24785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "9b83adfdc9fe47b2b59f6ac3849f749e"
format_stamp: "Formatted at 2026-09-24 15:27:04 on dist-test-slave-l766"
server_key: "570e161e72e3a71c76ce8ca819cad65c"
server_key_iv: "476afe28c47473c0348bd154cb4f686a"
server_key_version: "encryptionkey@0"
I20260924 15:27:05.000294 24785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "9b83adfdc9fe47b2b59f6ac3849f749e"
format_stamp: "Formatted at 2026-09-24 15:27:04 on dist-test-slave-l766"
server_key: "570e161e72e3a71c76ce8ca819cad65c"
server_key_iv: "476afe28c47473c0348bd154cb4f686a"
server_key_version: "encryptionkey@0"
I20260924 15:27:05.001592 24785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:05.002405 24800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:05.002583 24785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:05.002682 24785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "9b83adfdc9fe47b2b59f6ac3849f749e"
format_stamp: "Formatted at 2026-09-24 15:27:04 on dist-test-slave-l766"
server_key: "570e161e72e3a71c76ce8ca819cad65c"
server_key_iv: "476afe28c47473c0348bd154cb4f686a"
server_key_version: "encryptionkey@0"
I20260924 15:27:05.002771 24785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:05.044221 24785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:05.046727 24785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:05.046902 24785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:05.050802 24785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35311
I20260924 15:27:05.050843 24853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35311 every 8 connection(s)
I20260924 15:27:05.051232 24785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:05.052251 24854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:05.054520 24854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e: Bootstrap starting.
I20260924 15:27:05.055038 24854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:05.055301 24854 log.cc:824] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:05.055933 24854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e: No bootstrap required, opened a new log
I20260924 15:27:05.057449 24854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83adfdc9fe47b2b59f6ac3849f749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } }
I20260924 15:27:05.057581 24854 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:05.057603 24854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b83adfdc9fe47b2b59f6ac3849f749e, State: Initialized, Role: FOLLOWER
I20260924 15:27:05.057756 24854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83adfdc9fe47b2b59f6ac3849f749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } }
I20260924 15:27:05.057880 24854 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:05.057904 24854 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:05.057932 24854 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:05.058389 24854 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83adfdc9fe47b2b59f6ac3849f749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } }
I20260924 15:27:05.058470 24854 leader_election.cc:302] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b83adfdc9fe47b2b59f6ac3849f749e; no voters:
I20260924 15:27:05.058585 24854 leader_election.cc:288] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:05.058632 24857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:05.058789 24857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [term 1 LEADER]: Becoming Leader. State: Replica: 9b83adfdc9fe47b2b59f6ac3849f749e, State: Running, Role: LEADER
I20260924 15:27:05.058825 24854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:05.058926 24857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83adfdc9fe47b2b59f6ac3849f749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } }
I20260924 15:27:05.059453 24858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b83adfdc9fe47b2b59f6ac3849f749e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83adfdc9fe47b2b59f6ac3849f749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } } }
I20260924 15:27:05.059541 24858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:05.059628 24859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b83adfdc9fe47b2b59f6ac3849f749e. Latest consensus state: current_term: 1 leader_uuid: "9b83adfdc9fe47b2b59f6ac3849f749e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b83adfdc9fe47b2b59f6ac3849f749e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35311 } } }
I20260924 15:27:05.059895 24859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:05.059902 24866 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:05.060456 24866 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:05.060874 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 24785
I20260924 15:27:05.060981 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:05.061172 4409 external_mini_cluster.cc:1428] Setting key 7d243c3458c98d365ce4a68233e0fc76
I20260924 15:27:05.062166 24866 catalog_manager.cc:1347] Generated new cluster ID: 5a6284ed9e7049f8b554dc96e31c7327
I20260924 15:27:05.062218 24866 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:05.071237 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:05.061621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40135 (local address 127.0.0.1:35311)
0924 15:27:05.061767 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:05.061777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:05.061827 (+ 50us) server_negotiation.cc:407] Connection header received
0924 15:27:05.061971 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:05.061979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:05.062034 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:05.065968 (+ 3934us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:05.066229 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:05.067899 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:05.069052 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:05.069639 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:05.069738 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:05.070527 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:05.070530 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:05.070531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:05.070741 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:05.070783 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:05.070797 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:05.070873 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:05.070973 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:27:05.070999 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260924 15:27:05.072181 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:05.075321 24866 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:05.075981 24866 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:05.077183 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:05.073643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37739 (local address 127.0.0.1:35311)
0924 15:27:05.073680 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:05.073682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:05.073881 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:27:05.073953 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:05.073954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:05.073965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:05.074017 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:05.074127 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:05.074813 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:05.075847 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:05.076174 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:05.076245 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:05.076935 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:05.076936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:05.076937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:05.076941 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:05.076956 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:05.076961 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:05.077022 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:05.077097 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:27:05.077109 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:27:05.080178 24866 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b83adfdc9fe47b2b59f6ac3849f749e: Generated new TSK 0
I20260924 15:27:05.080401 24866 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:05.083874 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 24785
W20260924 15:27:05.088354 24770 connection.cc:535] client connection to 127.0.0.1:35311 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35311): Connection reset by peer (error 104)
2026-09-24T15:27:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:05 dist-test-slave-l766 krb5kdc[24885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:05 dist-test-slave-l766 krb5kdc[24885](info): set up 2 sockets
Sep 24 15:27:05 dist-test-slave-l766 krb5kdc[24885](info): commencing operation
krb5kdc: starting...
W20260924 15:27:06.128814 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:06 dist-test-slave-l766 krb5kdc[24885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:06.245081 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:06.323745 24898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:06.323989 24898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:06.324026 24898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:06.325176 24898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:06.325251 24898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:06.325275 24898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:06.325294 24898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:06.325321 24898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:06.326854 24898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36583
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:06.327102 24898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:06.327314 24898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:06.332526 24903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:06.332527 24904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:06.332554 24906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:06.332779 24898 server_base.cc:1034] running on GCE node
I20260924 15:27:06.333014 24898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:06.333340 24898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:06.334522 24898 hybrid_clock.cc:648] HybridClock initialized: now 1790263626334497 us; error 46 us; skew 500 ppm
Sep 24 15:27:06 dist-test-slave-l766 krb5kdc[24885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263626, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:06.346156 24898 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:06.346616 24898 webserver.cc:460] Webserver started at http://127.0.0.1:42543/ using document root <none> and password file <none>
I20260924 15:27:06.346850 24898 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:06.346920 24898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:06.347072 24898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:06.348124 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "af8b03256c634623bce82a89d5009d29"
format_stamp: "Formatted at 2026-09-24 15:27:06 on dist-test-slave-l766"
server_key: "c957f2726986c7c4743e37458cf1c8d9"
server_key_iv: "c9baffb2fd735e5a236088587107a606"
server_key_version: "encryptionkey@0"
I20260924 15:27:06.348500 24898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "af8b03256c634623bce82a89d5009d29"
format_stamp: "Formatted at 2026-09-24 15:27:06 on dist-test-slave-l766"
server_key: "c957f2726986c7c4743e37458cf1c8d9"
server_key_iv: "c9baffb2fd735e5a236088587107a606"
server_key_version: "encryptionkey@0"
I20260924 15:27:06.350005 24898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:06.350757 24913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:06.350929 24898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:06.351035 24898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "af8b03256c634623bce82a89d5009d29"
format_stamp: "Formatted at 2026-09-24 15:27:06 on dist-test-slave-l766"
server_key: "c957f2726986c7c4743e37458cf1c8d9"
server_key_iv: "c9baffb2fd735e5a236088587107a606"
server_key_version: "encryptionkey@0"
I20260924 15:27:06.351109 24898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:06.354462 24898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:06.356698 24898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:06.356866 24898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:06.361073 24898 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36583
I20260924 15:27:06.361100 24966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36583 every 8 connection(s)
I20260924 15:27:06.361416 24898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:06.362413 24967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:06.364547 24967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29: Bootstrap starting.
I20260924 15:27:06.365087 24967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:06.365340 24967 log.cc:824] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:06.366043 24967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29: No bootstrap required, opened a new log
I20260924 15:27:06.367623 24967 raft_consensus.cc:348] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8b03256c634623bce82a89d5009d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260924 15:27:06.367755 24967 raft_consensus.cc:374] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:06.367779 24967 raft_consensus.cc:729] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af8b03256c634623bce82a89d5009d29, State: Initialized, Role: FOLLOWER
I20260924 15:27:06.367925 24967 consensus_queue.cc:260] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8b03256c634623bce82a89d5009d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260924 15:27:06.368003 24967 raft_consensus.cc:388] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:06.368026 24967 raft_consensus.cc:482] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:06.368053 24967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:06.368521 24967 raft_consensus.cc:504] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8b03256c634623bce82a89d5009d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260924 15:27:06.368602 24967 leader_election.cc:302] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af8b03256c634623bce82a89d5009d29; no voters:
I20260924 15:27:06.368721 24967 leader_election.cc:288] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:06.368800 24970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:06.368971 24967 sys_catalog.cc:564] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:06.369202 24970 raft_consensus.cc:686] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [term 1 LEADER]: Becoming Leader. State: Replica: af8b03256c634623bce82a89d5009d29, State: Running, Role: LEADER
I20260924 15:27:06.369380 24970 consensus_queue.cc:237] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8b03256c634623bce82a89d5009d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } }
I20260924 15:27:06.369447 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 24898
I20260924 15:27:06.369563 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:06.369772 4409 external_mini_cluster.cc:1428] Setting key e37dd85843acedee5e141d6fa6dbe2f3
I20260924 15:27:06.370020 24972 sys_catalog.cc:455] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af8b03256c634623bce82a89d5009d29. Latest consensus state: current_term: 1 leader_uuid: "af8b03256c634623bce82a89d5009d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8b03256c634623bce82a89d5009d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } } }
I20260924 15:27:06.370103 24972 sys_catalog.cc:458] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:06.370596 24971 sys_catalog.cc:455] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af8b03256c634623bce82a89d5009d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af8b03256c634623bce82a89d5009d29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36583 } } }
I20260924 15:27:06.370824 24971 sys_catalog.cc:458] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29 [sys.catalog]: This master's current role is: LEADER
W20260924 15:27:06.371061 24988 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:06.371117 24988 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:06.371187 24983 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:06.371374 24983 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:06.372767 24983 catalog_manager.cc:1347] Generated new cluster ID: 517ebe6fd6b04734b1d4e9ec45477684
I20260924 15:27:06.372828 24983 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:06.373117 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:06.370449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53655 (local address 127.0.0.1:36583)
0924 15:27:06.370878 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:06.370889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:06.370919 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:27:06.371034 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:06.371041 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:06.371082 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:06.371953 (+ 871us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:06.372687 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:06.372691 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:06.372691 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:06.372696 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:06.372736 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:06.372750 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:06.372814 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:06.372906 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 15:27:06.372946 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":71,"threads_started":1}
I20260924 15:27:06.374053 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:06.376772 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:06.375583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39393 (local address 127.0.0.1:36583)
0924 15:27:06.375625 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:06.375627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:06.375754 (+ 127us) server_negotiation.cc:407] Connection header received
0924 15:27:06.375851 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:06.375852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:06.375856 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:06.375884 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:06.376547 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:06.376549 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:06.376549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:06.376553 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:06.376566 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:06.376571 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:06.376601 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:06.376705 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 15:27:06.376716 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:27:06.383689 24983 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:06.383863 24983 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:06.392223 24983 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af8b03256c634623bce82a89d5009d29: Generated new TSK 0
I20260924 15:27:06.392431 24983 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:06.395148 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 24898
2026-09-24T15:27:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:06 dist-test-slave-l766 krb5kdc[24998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:06 dist-test-slave-l766 krb5kdc[24998](info): set up 2 sockets
Sep 24 15:27:06 dist-test-slave-l766 krb5kdc[24998](info): commencing operation
krb5kdc: starting...
W20260924 15:27:07.428864 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:07 dist-test-slave-l766 krb5kdc[24998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:07.545025 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:07.627717 25011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:07.627957 25011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:07.627996 25011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:07.629187 25011 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:07.629261 25011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:07.629287 25011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:07.629318 25011 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:07.629345 25011 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:07.630941 25011 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:07.631218 25011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:07.631464 25011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:07.636371 25016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:07.636445 25011 server_base.cc:1034] running on GCE node
W20260924 15:27:07.636366 25019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:07.636493 25017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:07.636785 25011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:07.637082 25011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:07.638230 25011 hybrid_clock.cc:648] HybridClock initialized: now 1790263627638210 us; error 34 us; skew 500 ppm
Sep 24 15:27:07 dist-test-slave-l766 krb5kdc[24998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263627, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:07.640730 25011 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:07.641139 25011 webserver.cc:460] Webserver started at http://127.0.0.1:38903/ using document root <none> and password file <none>
I20260924 15:27:07.641389 25011 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:07.641588 25011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:07.641758 25011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:07.642877 25011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "89f37334f3784828832555c66072fe5a"
format_stamp: "Formatted at 2026-09-24 15:27:07 on dist-test-slave-l766"
server_key: "2acd5c5255caf0b82c9b7c19e1be4a07"
server_key_iv: "5632b69a3a49004ceef1522be7efd5ee"
server_key_version: "encryptionkey@0"
I20260924 15:27:07.643239 25011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "89f37334f3784828832555c66072fe5a"
format_stamp: "Formatted at 2026-09-24 15:27:07 on dist-test-slave-l766"
server_key: "2acd5c5255caf0b82c9b7c19e1be4a07"
server_key_iv: "5632b69a3a49004ceef1522be7efd5ee"
server_key_version: "encryptionkey@0"
I20260924 15:27:07.644565 25011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:07.645321 25026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:07.645601 25011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:07.645707 25011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "89f37334f3784828832555c66072fe5a"
format_stamp: "Formatted at 2026-09-24 15:27:07 on dist-test-slave-l766"
server_key: "2acd5c5255caf0b82c9b7c19e1be4a07"
server_key_iv: "5632b69a3a49004ceef1522be7efd5ee"
server_key_version: "encryptionkey@0"
I20260924 15:27:07.645802 25011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:07.654592 25011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:07.656838 25011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:07.657001 25011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:07.660955 25011 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36093
I20260924 15:27:07.661007 25078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36093 every 8 connection(s)
I20260924 15:27:07.661370 25011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:07.662443 25079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:07.664688 25079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a: Bootstrap starting.
I20260924 15:27:07.665266 25079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:07.665551 25079 log.cc:824] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:07.666338 25079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a: No bootstrap required, opened a new log
I20260924 15:27:07.667904 25079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f37334f3784828832555c66072fe5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260924 15:27:07.668035 25079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:07.668058 25079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f37334f3784828832555c66072fe5a, State: Initialized, Role: FOLLOWER
I20260924 15:27:07.668210 25079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f37334f3784828832555c66072fe5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260924 15:27:07.668273 25079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:07.668294 25079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:07.668342 25079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:07.668870 25079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f37334f3784828832555c66072fe5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260924 15:27:07.668982 25079 leader_election.cc:302] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89f37334f3784828832555c66072fe5a; no voters:
I20260924 15:27:07.669186 25079 leader_election.cc:288] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:07.669250 25082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:07.669502 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25011
I20260924 15:27:07.669611 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:07.669793 4409 external_mini_cluster.cc:1428] Setting key 00e776787fe0da9206b15633cb94602d
I20260924 15:27:07.669433 25079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:07.670042 25082 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [term 1 LEADER]: Becoming Leader. State: Replica: 89f37334f3784828832555c66072fe5a, State: Running, Role: LEADER
I20260924 15:27:07.670135 25082 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f37334f3784828832555c66072fe5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } }
I20260924 15:27:07.671727 25084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89f37334f3784828832555c66072fe5a. Latest consensus state: current_term: 1 leader_uuid: "89f37334f3784828832555c66072fe5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f37334f3784828832555c66072fe5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } } }
I20260924 15:27:07.671815 25084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [sys.catalog]: This master's current role is: LEADER
W20260924 15:27:07.672004 25098 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:07.672065 25098 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:07.672170 25083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89f37334f3784828832555c66072fe5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89f37334f3784828832555c66072fe5a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36093 } } }
I20260924 15:27:07.672175 25101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:07.672283 25083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:07.672495 25101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:07.674417 25101 catalog_manager.cc:1347] Generated new cluster ID: a4ce60ee25c24b5c9f8e1cc34daebb7d
I20260924 15:27:07.674471 25101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:07 dist-test-slave-l766 krb5kdc[24998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:27:07.680488 25101 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:07.681249 25101 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:07.682446 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:07.670511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53197 (local address 127.0.0.1:36093)
0924 15:27:07.670741 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:07.670751 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:07.670788 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:27:07.670908 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:07.670915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:07.670970 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:07.671117 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:07.671933 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:07.673332 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:07.674358 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:07.674934 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:07.675045 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:07.677894 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:07.677899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:07.677899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:07.677911 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:07.680336 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:07.680765 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:07.680769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:07.680769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:07.680879 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:07.681089 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:07.681090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:07.681090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:07.681368 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:07.681535 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:07.682079 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 15:27:07.682233 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":83,"threads_started":1}
I20260924 15:27:07.683307 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:07.686728 25101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89f37334f3784828832555c66072fe5a: Generated new TSK 0
I20260924 15:27:07.686954 25101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:07.691797 25091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:07.684710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52861 (local address 127.0.0.1:36093)
0924 15:27:07.684753 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:07.684755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:07.684842 (+ 87us) server_negotiation.cc:407] Connection header received
0924 15:27:07.684885 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:07.684886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:07.684895 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:07.685099 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:07.685696 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:07.686480 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:07.687610 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:07.687799 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:07.687863 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:07.689277 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:07.689278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:07.689278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:07.689284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:07.690743 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:07.691139 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:07.691140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:07.691140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:07.691187 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:07.691391 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:07.691392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:07.691392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:07.691480 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:07.691547 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:07.691684 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 15:27:07.691741 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:27:07.693960 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25011
2026-09-24T15:27:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:07 dist-test-slave-l766 krb5kdc[25110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:07 dist-test-slave-l766 krb5kdc[25110](info): set up 2 sockets
Sep 24 15:27:07 dist-test-slave-l766 krb5kdc[25110](info): commencing operation
krb5kdc: starting...
W20260924 15:27:08.724773 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:08 dist-test-slave-l766 krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:08.838819 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:08.919896 25123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:08.920142 25123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:08.920181 25123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:08.921312 25123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:08.921386 25123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:08.921411 25123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:08.921432 25123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:08.921460 25123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:08.922916 25123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:08.923182 25123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:08.923408 25123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:08.928157 25131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:08.928188 25129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:08.928162 25128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:08.928454 25123 server_base.cc:1034] running on GCE node
I20260924 15:27:08.928681 25123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:08.928951 25123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:08.930145 25123 hybrid_clock.cc:648] HybridClock initialized: now 1790263628930129 us; error 34 us; skew 500 ppm
Sep 24 15:27:08 dist-test-slave-l766 krb5kdc[25110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263628, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:08.932583 25123 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:08.933044 25123 webserver.cc:460] Webserver started at http://127.0.0.1:37883/ using document root <none> and password file <none>
I20260924 15:27:08.933275 25123 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:08.933348 25123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:08.933490 25123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:08.934566 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "61483929d67440aabc74bc8b6b363ab8"
format_stamp: "Formatted at 2026-09-24 15:27:08 on dist-test-slave-l766"
server_key: "790343e8ee801fbc025c00257a8ddcc6"
server_key_iv: "1fb74d5c884625dac48673106adef69c"
server_key_version: "encryptionkey@0"
I20260924 15:27:08.934916 25123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "61483929d67440aabc74bc8b6b363ab8"
format_stamp: "Formatted at 2026-09-24 15:27:08 on dist-test-slave-l766"
server_key: "790343e8ee801fbc025c00257a8ddcc6"
server_key_iv: "1fb74d5c884625dac48673106adef69c"
server_key_version: "encryptionkey@0"
I20260924 15:27:08.936190 25123 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:27:08.936889 25138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:08.937062 25123 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:08.937150 25123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "61483929d67440aabc74bc8b6b363ab8"
format_stamp: "Formatted at 2026-09-24 15:27:08 on dist-test-slave-l766"
server_key: "790343e8ee801fbc025c00257a8ddcc6"
server_key_iv: "1fb74d5c884625dac48673106adef69c"
server_key_version: "encryptionkey@0"
I20260924 15:27:08.937224 25123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:08.958768 25123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:08.960686 25123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:08.960853 25123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:08.965159 25123 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41327
I20260924 15:27:08.965210 25190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41327 every 8 connection(s)
I20260924 15:27:08.965588 25123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:08.966554 25191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:08.968839 25191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8: Bootstrap starting.
I20260924 15:27:08.969377 25191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:08.969604 25191 log.cc:824] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:08.970247 25191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8: No bootstrap required, opened a new log
I20260924 15:27:08.971764 25191 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61483929d67440aabc74bc8b6b363ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } }
I20260924 15:27:08.971912 25191 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:08.971935 25191 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61483929d67440aabc74bc8b6b363ab8, State: Initialized, Role: FOLLOWER
I20260924 15:27:08.972034 25191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61483929d67440aabc74bc8b6b363ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } }
I20260924 15:27:08.972085 25191 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:08.972105 25191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:08.972131 25191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:08.972697 25191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61483929d67440aabc74bc8b6b363ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } }
I20260924 15:27:08.972878 25191 leader_election.cc:302] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61483929d67440aabc74bc8b6b363ab8; no voters:
I20260924 15:27:08.973068 25191 leader_election.cc:288] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:08.973345 25191 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:08.973400 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25123
I20260924 15:27:08.973527 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:08.973731 4409 external_mini_cluster.cc:1428] Setting key 532969c2c4aa359628762a0f50a7f6ec
I20260924 15:27:08.973127 25194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:08.974128 25194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [term 1 LEADER]: Becoming Leader. State: Replica: 61483929d67440aabc74bc8b6b363ab8, State: Running, Role: LEADER
I20260924 15:27:08.974290 25194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61483929d67440aabc74bc8b6b363ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } }
I20260924 15:27:08.974695 25201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61483929d67440aabc74bc8b6b363ab8. Latest consensus state: current_term: 1 leader_uuid: "61483929d67440aabc74bc8b6b363ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61483929d67440aabc74bc8b6b363ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } } }
I20260924 15:27:08.974790 25201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:08.974861 25195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61483929d67440aabc74bc8b6b363ab8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61483929d67440aabc74bc8b6b363ab8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41327 } } }
I20260924 15:27:08.974922 25195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8 [sys.catalog]: This master's current role is: LEADER
W20260924 15:27:08.975231 25210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:08.975291 25210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:08.975346 25211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:08.975492 25211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:08.976827 25211 catalog_manager.cc:1347] Generated new cluster ID: d743627fe1be49a9aff0fb58abab0f41
I20260924 15:27:08.976881 25211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:08.979998 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:08.974223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52683 (local address 127.0.0.1:41327)
0924 15:27:08.974397 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:08.974407 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:08.974443 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:27:08.974669 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:08.974676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:08.974739 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:08.975207 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:08.975321 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:08.976701 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:08.977868 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:08.978459 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:08.978571 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:08.979277 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:08.979289 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:08.979289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:08.979511 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:08.979549 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:08.979564 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:08.979636 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:08.979710 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 15:27:08.979772 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260924 15:27:08.981017 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:08.986014 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:08.982562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58551 (local address 127.0.0.1:41327)
0924 15:27:08.982595 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:08.982597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:08.982775 (+ 178us) server_negotiation.cc:407] Connection header received
0924 15:27:08.982840 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:08.982841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:08.982850 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:08.982899 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:08.983041 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:08.983728 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:08.984745 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:08.985032 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:08.985091 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:08.985748 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:08.985749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:08.985749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:08.985754 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:08.985767 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:08.985770 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:08.985861 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:08.985923 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:27:08.985944 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:27:09.019126 25211 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:09.019706 25211 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:09.024399 25211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61483929d67440aabc74bc8b6b363ab8: Generated new TSK 0
I20260924 15:27:09.024600 25211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:09.028591 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25123
2026-09-24T15:27:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:09 dist-test-slave-l766 krb5kdc[25222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:09 dist-test-slave-l766 krb5kdc[25222](info): set up 2 sockets
Sep 24 15:27:09 dist-test-slave-l766 krb5kdc[25222](info): commencing operation
krb5kdc: starting...
W20260924 15:27:10.069132 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:10 dist-test-slave-l766 krb5kdc[25222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263630, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:10.183954 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:10.264264 25235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:10.264468 25235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:10.264492 25235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:10.265563 25235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:10.265599 25235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:10.265612 25235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:10.265625 25235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:10.265636 25235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:10.267100 25235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:10.267298 25235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:10.267464 25235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:10.272392 25243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:10.272403 25241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:10.272403 25240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:10.272471 25235 server_base.cc:1034] running on GCE node
I20260924 15:27:10.272909 25235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:10.273195 25235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:10.274340 25235 hybrid_clock.cc:648] HybridClock initialized: now 1790263630274328 us; error 29 us; skew 500 ppm
Sep 24 15:27:10 dist-test-slave-l766 krb5kdc[25222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263630, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:10.276845 25235 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:10.277315 25235 webserver.cc:460] Webserver started at http://127.0.0.1:42895/ using document root <none> and password file <none>
I20260924 15:27:10.277558 25235 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:10.277603 25235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:10.277771 25235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:10.278851 25235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "1772ab3389c64204b6b22b57547561ef"
format_stamp: "Formatted at 2026-09-24 15:27:10 on dist-test-slave-l766"
server_key: "e5f2cd1759effa5c9cab1d8cd8f3076e"
server_key_iv: "1bff1ea82962decc04ff99d4231360e7"
server_key_version: "encryptionkey@0"
I20260924 15:27:10.279206 25235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "1772ab3389c64204b6b22b57547561ef"
format_stamp: "Formatted at 2026-09-24 15:27:10 on dist-test-slave-l766"
server_key: "e5f2cd1759effa5c9cab1d8cd8f3076e"
server_key_iv: "1bff1ea82962decc04ff99d4231360e7"
server_key_version: "encryptionkey@0"
I20260924 15:27:10.280501 25235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:10.281272 25250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:10.281477 25235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:10.281571 25235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "1772ab3389c64204b6b22b57547561ef"
format_stamp: "Formatted at 2026-09-24 15:27:10 on dist-test-slave-l766"
server_key: "e5f2cd1759effa5c9cab1d8cd8f3076e"
server_key_iv: "1bff1ea82962decc04ff99d4231360e7"
server_key_version: "encryptionkey@0"
I20260924 15:27:10.281649 25235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:10.284904 25235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:10.286798 25235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:10.286950 25235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:10.290834 25235 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42633
I20260924 15:27:10.290905 25302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42633 every 8 connection(s)
I20260924 15:27:10.291188 25235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:10.292138 25303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:10.294397 25303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef: Bootstrap starting.
I20260924 15:27:10.294961 25303 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:10.295236 25303 log.cc:824] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:10.295897 25303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef: No bootstrap required, opened a new log
I20260924 15:27:10.297420 25303 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772ab3389c64204b6b22b57547561ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } }
I20260924 15:27:10.297550 25303 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:10.297572 25303 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1772ab3389c64204b6b22b57547561ef, State: Initialized, Role: FOLLOWER
I20260924 15:27:10.297748 25303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772ab3389c64204b6b22b57547561ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } }
I20260924 15:27:10.297833 25303 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:10.297857 25303 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:10.297930 25303 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:10.298441 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25235
I20260924 15:27:10.298460 25303 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772ab3389c64204b6b22b57547561ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } }
I20260924 15:27:10.298529 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:10.298573 25303 leader_election.cc:302] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1772ab3389c64204b6b22b57547561ef; no voters:
I20260924 15:27:10.298779 4409 external_mini_cluster.cc:1428] Setting key cfd8e73d73c5d076b68137a6f2d92d44
I20260924 15:27:10.298774 25303 leader_election.cc:288] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:10.298803 25306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:10.298925 25306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [term 1 LEADER]: Becoming Leader. State: Replica: 1772ab3389c64204b6b22b57547561ef, State: Running, Role: LEADER
I20260924 15:27:10.299072 25303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:10.299063 25306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772ab3389c64204b6b22b57547561ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } }
I20260924 15:27:10.299525 25308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1772ab3389c64204b6b22b57547561ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772ab3389c64204b6b22b57547561ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } } }
I20260924 15:27:10.299539 25307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1772ab3389c64204b6b22b57547561ef. Latest consensus state: current_term: 1 leader_uuid: "1772ab3389c64204b6b22b57547561ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1772ab3389c64204b6b22b57547561ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42633 } } }
I20260924 15:27:10.299680 25308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:10.299791 25307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:10.300972 25312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:10.301503 25312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:10.301558 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:10.299338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41157 (local address 127.0.0.1:42633)
0924 15:27:10.299519 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:10.299530 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:10.299563 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:27:10.299676 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:10.299685 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:10.299714 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:10.299853 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:10.300981 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:10.300986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:10.300986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:10.300996 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:10.301108 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:10.301124 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:10.301192 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:10.301341 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 15:27:10.301385 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260924 15:27:10.302786 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:10.305284 25310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:10.304350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51787 (local address 127.0.0.1:42633)
0924 15:27:10.304386 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:10.304387 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:10.304509 (+ 122us) server_negotiation.cc:407] Connection header received
0924 15:27:10.304562 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:10.304563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:10.304567 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:10.304604 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:10.305082 (+ 478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:10.305083 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:10.305083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:10.305086 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:10.305098 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:10.305101 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:10.305139 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:10.305193 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 15:27:10.305202 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:27:10.306229 25312 catalog_manager.cc:1347] Generated new cluster ID: 02d92677b2594cd3b5e834c73383fa6c
I20260924 15:27:10.306286 25312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:10.318809 25312 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:10.318974 25312 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:10.322202 25312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1772ab3389c64204b6b22b57547561ef: Generated new TSK 0
I20260924 15:27:10.322397 25312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:10.327025 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25235
2026-09-24T15:27:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:10 dist-test-slave-l766 krb5kdc[25334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:10 dist-test-slave-l766 krb5kdc[25334](info): set up 2 sockets
Sep 24 15:27:10 dist-test-slave-l766 krb5kdc[25334](info): commencing operation
krb5kdc: starting...
W20260924 15:27:11.352847 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:11 dist-test-slave-l766 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:11.469287 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:11.551615 25347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:11.551819 25347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:11.551841 25347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:11.552987 25347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:11.553031 25347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:11.553045 25347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:11.553083 25347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:11.553109 25347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:11.554644 25347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:11.554901 25347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:11.555135 25347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:11.560083 25353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:11.560092 25352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:11.560349 25355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:11.560300 25347 server_base.cc:1034] running on GCE node
I20260924 15:27:11.560588 25347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:11.560895 25347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:11.562098 25347 hybrid_clock.cc:648] HybridClock initialized: now 1790263631562076 us; error 39 us; skew 500 ppm
Sep 24 15:27:11 dist-test-slave-l766 krb5kdc[25334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:11.573975 25347 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:11.574467 25347 webserver.cc:460] Webserver started at http://127.0.0.1:33037/ using document root <none> and password file <none>
I20260924 15:27:11.574682 25347 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:11.574725 25347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:11.574818 25347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:11.575832 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e65a055f04ec4da985da886dac2cf692"
format_stamp: "Formatted at 2026-09-24 15:27:11 on dist-test-slave-l766"
server_key: "21021043056fd28dec61b83d3be1ffb9"
server_key_iv: "2e60a43a6bd334420e0f798ca26f7fb8"
server_key_version: "encryptionkey@0"
I20260924 15:27:11.576131 25347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e65a055f04ec4da985da886dac2cf692"
format_stamp: "Formatted at 2026-09-24 15:27:11 on dist-test-slave-l766"
server_key: "21021043056fd28dec61b83d3be1ffb9"
server_key_iv: "2e60a43a6bd334420e0f798ca26f7fb8"
server_key_version: "encryptionkey@0"
I20260924 15:27:11.577385 25347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:11.578229 25362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:11.578465 25347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:11.578554 25347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e65a055f04ec4da985da886dac2cf692"
format_stamp: "Formatted at 2026-09-24 15:27:11 on dist-test-slave-l766"
server_key: "21021043056fd28dec61b83d3be1ffb9"
server_key_iv: "2e60a43a6bd334420e0f798ca26f7fb8"
server_key_version: "encryptionkey@0"
I20260924 15:27:11.578610 25347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:11.592396 25347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:11.594239 25347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:11.594354 25347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:11.598234 25347 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39019
I20260924 15:27:11.598285 25414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39019 every 8 connection(s)
I20260924 15:27:11.598618 25347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:11.599622 25415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:11.601882 25415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692: Bootstrap starting.
I20260924 15:27:11.602459 25415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:11.602733 25415 log.cc:824] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:11.603415 25415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692: No bootstrap required, opened a new log
I20260924 15:27:11.604943 25415 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65a055f04ec4da985da886dac2cf692" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260924 15:27:11.605100 25415 raft_consensus.cc:374] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:11.605149 25415 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65a055f04ec4da985da886dac2cf692, State: Initialized, Role: FOLLOWER
I20260924 15:27:11.605295 25415 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65a055f04ec4da985da886dac2cf692" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260924 15:27:11.605376 25415 raft_consensus.cc:388] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:11.605424 25415 raft_consensus.cc:482] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:11.605475 25415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:11.605738 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25347
I20260924 15:27:11.605898 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:11.606113 4409 external_mini_cluster.cc:1428] Setting key 0b283a692f45f8a7c64b921711cbd593
I20260924 15:27:11.606225 25415 raft_consensus.cc:504] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65a055f04ec4da985da886dac2cf692" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260924 15:27:11.606333 25415 leader_election.cc:302] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e65a055f04ec4da985da886dac2cf692; no voters:
I20260924 15:27:11.606489 25415 leader_election.cc:288] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:11.606518 25418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:11.606719 25418 raft_consensus.cc:686] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [term 1 LEADER]: Becoming Leader. State: Replica: e65a055f04ec4da985da886dac2cf692, State: Running, Role: LEADER
I20260924 15:27:11.606814 25415 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:11.606829 25418 consensus_queue.cc:237] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65a055f04ec4da985da886dac2cf692" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } }
I20260924 15:27:11.607203 25420 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e65a055f04ec4da985da886dac2cf692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65a055f04ec4da985da886dac2cf692" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } } }
I20260924 15:27:11.607219 25421 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e65a055f04ec4da985da886dac2cf692. Latest consensus state: current_term: 1 leader_uuid: "e65a055f04ec4da985da886dac2cf692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e65a055f04ec4da985da886dac2cf692" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39019 } } }
I20260924 15:27:11.607306 25420 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:11.607313 25421 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:11.607978 25424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:11.608532 25424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:11.610245 25424 catalog_manager.cc:1347] Generated new cluster ID: 9ca54765d3434cffb3b9c2bb5b617024
I20260924 15:27:11.610298 25424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:11 dist-test-slave-l766 krb5kdc[25334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263631, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:27:11.618574 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:11.606635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:42655 (local address 127.0.0.1:39019)
0924 15:27:11.607212 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:11.607231 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:11.607266 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:27:11.607393 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:11.607401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:11.607466 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:11.607604 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:11.608418 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:11.609611 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:11.610688 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:11.611272 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:11.611382 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:11.614216 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:11.614221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:11.614221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:11.614232 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:11.616786 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:11.617259 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:11.617263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:11.617263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:11.617384 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:11.617625 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:11.617626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:11.617626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:11.617968 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:11.618120 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:11.618258 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 15:27:11.618364 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":80,"threads_started":1}
I20260924 15:27:11.619581 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:11.621903 25424 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:11.622537 25424 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:11.625530 25424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e65a055f04ec4da985da886dac2cf692: Generated new TSK 0
I20260924 15:27:11.625728 25424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:11.627961 25422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:11.621154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:50051 (local address 127.0.0.1:39019)
0924 15:27:11.621203 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:11.621204 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:11.621213 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:27:11.621259 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:11.621259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:11.621268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:11.621319 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:11.621993 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:11.622676 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:11.623687 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:11.623997 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:11.624059 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:11.625455 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:11.625457 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:11.625457 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:11.625463 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:11.626856 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:11.627266 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:11.627266 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:11.627267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:11.627313 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:11.627555 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:11.627556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:11.627556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:11.627647 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:11.627716 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:11.627862 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 15:27:11.627908 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:11.629856 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25347
W20260924 15:27:11.633688 25332 connection.cc:535] client connection to 127.0.0.1:39019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39019): Connection reset by peer (error 104)
2026-09-24T15:27:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:11 dist-test-slave-l766 krb5kdc[25446](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:11 dist-test-slave-l766 krb5kdc[25446](info): set up 2 sockets
Sep 24 15:27:11 dist-test-slave-l766 krb5kdc[25446](info): commencing operation
krb5kdc: starting...
W20260924 15:27:12.660857 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:12 dist-test-slave-l766 krb5kdc[25446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:12.775101 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:12.854038 25459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:12.854287 25459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:12.854329 25459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:12.855520 25459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:12.855592 25459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:12.855615 25459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:12.855638 25459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:12.855666 25459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:12.857111 25459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:12.857409 25459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:12.857632 25459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:12.862442 25465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:12.862460 25464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:12.862460 25467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:12.862752 25459 server_base.cc:1034] running on GCE node
I20260924 15:27:12.863030 25459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:12.863303 25459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:12.864475 25459 hybrid_clock.cc:648] HybridClock initialized: now 1790263632864465 us; error 28 us; skew 500 ppm
Sep 24 15:27:12 dist-test-slave-l766 krb5kdc[25446](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:12.866976 25459 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:12.867417 25459 webserver.cc:460] Webserver started at http://127.0.0.1:43155/ using document root <none> and password file <none>
I20260924 15:27:12.867656 25459 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:12.867725 25459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:12.867874 25459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:12.868904 25459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "43799059e7764229a7cbd92016a88721"
format_stamp: "Formatted at 2026-09-24 15:27:12 on dist-test-slave-l766"
server_key: "c70bc10dfbc9d377a436c2aa3db24fc4"
server_key_iv: "49846468c47d443c0e94254eb1a1a15f"
server_key_version: "encryptionkey@0"
I20260924 15:27:12.869341 25459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "43799059e7764229a7cbd92016a88721"
format_stamp: "Formatted at 2026-09-24 15:27:12 on dist-test-slave-l766"
server_key: "c70bc10dfbc9d377a436c2aa3db24fc4"
server_key_iv: "49846468c47d443c0e94254eb1a1a15f"
server_key_version: "encryptionkey@0"
I20260924 15:27:12.870718 25459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:12.871428 25474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:12.871601 25459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:12.871706 25459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "43799059e7764229a7cbd92016a88721"
format_stamp: "Formatted at 2026-09-24 15:27:12 on dist-test-slave-l766"
server_key: "c70bc10dfbc9d377a436c2aa3db24fc4"
server_key_iv: "49846468c47d443c0e94254eb1a1a15f"
server_key_version: "encryptionkey@0"
I20260924 15:27:12.871779 25459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:12.882043 25459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:12.884482 25459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:12.884644 25459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:12.888667 25459 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35551
I20260924 15:27:12.888701 25526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35551 every 8 connection(s)
I20260924 15:27:12.889043 25459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:12.889361 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25459
I20260924 15:27:12.889487 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:12.889679 4409 external_mini_cluster.cc:1428] Setting key ed21eb27d1e3f95d8e1ce880179865ee
I20260924 15:27:12.890354 25527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:12.893275 25527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721: Bootstrap starting.
I20260924 15:27:12.893980 25527 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:12.894331 25527 log.cc:824] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:12.895052 25527 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721: No bootstrap required, opened a new log
I20260924 15:27:12.895174 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:12.890205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:56217 (local address 127.0.0.1:35551)
0924 15:27:12.890394 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:12.890422 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:12.890457 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:27:12.890903 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:12.890909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:12.890957 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:12.891064 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:12.891308 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:12.892160 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:12.893259 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:12.893699 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:12.893776 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:12.894485 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:12.894489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:12.894489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:12.894643 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:12.894672 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:12.894682 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:12.894745 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:12.894877 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 15:27:12.894920 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260924 15:27:12.896341 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:12.897249 25527 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43799059e7764229a7cbd92016a88721" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } }
I20260924 15:27:12.897390 25527 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:12.897439 25527 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43799059e7764229a7cbd92016a88721, State: Initialized, Role: FOLLOWER
I20260924 15:27:12.897629 25527 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43799059e7764229a7cbd92016a88721" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } }
I20260924 15:27:12.897763 25527 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:12.897835 25527 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:12.897894 25527 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:12.898428 25527 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43799059e7764229a7cbd92016a88721" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } }
I20260924 15:27:12.898518 25527 leader_election.cc:302] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43799059e7764229a7cbd92016a88721; no voters:
I20260924 15:27:12.898696 25527 leader_election.cc:288] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:12.898727 25537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:12.898890 25537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [term 1 LEADER]: Becoming Leader. State: Replica: 43799059e7764229a7cbd92016a88721, State: Running, Role: LEADER
I20260924 15:27:12.898952 25527 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:12.899024 25537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43799059e7764229a7cbd92016a88721" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } }
I20260924 15:27:12.899417 25537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43799059e7764229a7cbd92016a88721. Latest consensus state: current_term: 1 leader_uuid: "43799059e7764229a7cbd92016a88721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43799059e7764229a7cbd92016a88721" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } } }
I20260924 15:27:12.899572 25537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [sys.catalog]: This master's current role is: LEADER
W20260924 15:27:12.900732 25552 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:12.900836 25552 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:12.900899 25546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:12.901063 25546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:12.901612 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:12.897740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:54759 (local address 127.0.0.1:35551)
0924 15:27:12.897776 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:12.897778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:12.897963 (+ 185us) server_negotiation.cc:407] Connection header received
0924 15:27:12.897983 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:12.897984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:12.897994 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:12.898059 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:12.898207 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:12.898838 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:12.899851 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:12.900096 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:12.900426 (+ 330us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:12.901213 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:12.901215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:12.901215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:12.901220 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:12.901234 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:12.901238 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:12.901476 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:12.901507 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 15:27:12.901519 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:12.901878 25538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43799059e7764229a7cbd92016a88721" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43799059e7764229a7cbd92016a88721" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35551 } } }
I20260924 15:27:12.901996 25538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:12.903023 25554 mvcc.cc:205] Tried to move back new op lower bound from 7332919840368209920 to 7332919840355082240. Current Snapshot: MvccSnapshot[applied={T|T < 7332919840368209920}]
I20260924 15:27:12.903652 25546 catalog_manager.cc:1347] Generated new cluster ID: 36db61eee6a746fb937b9986c0ac8ce9
I20260924 15:27:12.903717 25546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:12.924124 25546 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:12.924710 25546 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:12.931058 25546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43799059e7764229a7cbd92016a88721: Generated new TSK 0
I20260924 15:27:12.931257 25546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:12.939636 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25459
W20260924 15:27:12.944187 25444 connection.cc:535] client connection to 127.0.0.1:35551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35551): Connection reset by peer (error 104)
2026-09-24T15:27:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:12 dist-test-slave-l766 krb5kdc[25558](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:12 dist-test-slave-l766 krb5kdc[25558](info): set up 2 sockets
Sep 24 15:27:12 dist-test-slave-l766 krb5kdc[25558](info): commencing operation
krb5kdc: starting...
W20260924 15:27:13.976926 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:14 dist-test-slave-l766 krb5kdc[25558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263634, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:14.092247 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:14.172349 25571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:14.172551 25571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:14.172575 25571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:14.173728 25571 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:14.173770 25571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:14.173784 25571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:14.173799 25571 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:14.173833 25571 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:14.175339 25571 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43397
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:14.175589 25571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:14.175810 25571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:14.180681 25579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:14.180717 25577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:14.180830 25576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:14.180717 25571 server_base.cc:1034] running on GCE node
I20260924 15:27:14.181160 25571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:14.181388 25571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:14.182550 25571 hybrid_clock.cc:648] HybridClock initialized: now 1790263634182536 us; error 27 us; skew 500 ppm
Sep 24 15:27:14 dist-test-slave-l766 krb5kdc[25558](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:14.184960 25571 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:14.185395 25571 webserver.cc:460] Webserver started at http://127.0.0.1:46109/ using document root <none> and password file <none>
I20260924 15:27:14.185621 25571 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:14.185664 25571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:14.185801 25571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:14.186895 25571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614"
format_stamp: "Formatted at 2026-09-24 15:27:14 on dist-test-slave-l766"
server_key: "d01f2970475b7ed617afaafdae5fbf9b"
server_key_iv: "a22289795ce83a4964781a8a49c2734a"
server_key_version: "encryptionkey@0"
I20260924 15:27:14.187265 25571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614"
format_stamp: "Formatted at 2026-09-24 15:27:14 on dist-test-slave-l766"
server_key: "d01f2970475b7ed617afaafdae5fbf9b"
server_key_iv: "a22289795ce83a4964781a8a49c2734a"
server_key_version: "encryptionkey@0"
I20260924 15:27:14.188583 25571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:14.189320 25586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:14.189514 25571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:14.189611 25571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614"
format_stamp: "Formatted at 2026-09-24 15:27:14 on dist-test-slave-l766"
server_key: "d01f2970475b7ed617afaafdae5fbf9b"
server_key_iv: "a22289795ce83a4964781a8a49c2734a"
server_key_version: "encryptionkey@0"
I20260924 15:27:14.189682 25571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:14.192847 25571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:14.194694 25571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:14.194809 25571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:14.198653 25571 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43397
I20260924 15:27:14.198700 25638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43397 every 8 connection(s)
I20260924 15:27:14.199024 25571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:14.199929 25639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:14.202138 25639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614: Bootstrap starting.
I20260924 15:27:14.202682 25639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:14.202977 25639 log.cc:824] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:14.203635 25639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614: No bootstrap required, opened a new log
I20260924 15:27:14.205158 25639 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260924 15:27:14.205309 25639 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:14.205348 25639 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7b762b3c7fc4b05ab61d44ff0cd5614, State: Initialized, Role: FOLLOWER
I20260924 15:27:14.205533 25639 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260924 15:27:14.205637 25639 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:14.205682 25639 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:14.205737 25639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:14.206266 25639 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260924 15:27:14.206395 25639 leader_election.cc:302] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7b762b3c7fc4b05ab61d44ff0cd5614; no voters:
I20260924 15:27:14.206571 25639 leader_election.cc:288] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:14.206611 25642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:14.206818 25639 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:14.206815 25642 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [term 1 LEADER]: Becoming Leader. State: Replica: e7b762b3c7fc4b05ab61d44ff0cd5614, State: Running, Role: LEADER
I20260924 15:27:14.206979 25642 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } }
I20260924 15:27:14.207046 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25571
I20260924 15:27:14.207170 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:14.207388 4409 external_mini_cluster.cc:1428] Setting key fa35035a6d7154fc3d8580d7847595b1
I20260924 15:27:14.207731 25644 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7b762b3c7fc4b05ab61d44ff0cd5614. Latest consensus state: current_term: 1 leader_uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260924 15:27:14.207803 25644 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:14.208403 25643 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7b762b3c7fc4b05ab61d44ff0cd5614" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43397 } } }
I20260924 15:27:14.208546 25643 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614 [sys.catalog]: This master's current role is: LEADER
W20260924 15:27:14.208766 25660 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:14.208824 25660 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:14.208874 25652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:14.209053 25652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:14.210330 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:14.207995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:58903 (local address 127.0.0.1:43397)
0924 15:27:14.208148 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:14.208159 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:14.208456 (+ 297us) server_negotiation.cc:407] Connection header received
0924 15:27:14.208644 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:14.208651 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:14.208682 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:14.208811 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:14.209880 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:14.209883 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:14.209884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:14.209888 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:14.209931 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:14.209945 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:14.210017 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:14.210107 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:27:14.210147 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":70,"threads_started":1}
I20260924 15:27:14.210554 25652 catalog_manager.cc:1347] Generated new cluster ID: c82a37625bcb4ad39020c790feefbce0
I20260924 15:27:14.210600 25652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:14.214231 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:14.216975 25657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:14.215905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:47507 (local address 127.0.0.1:43397)
0924 15:27:14.215939 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:14.215940 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:14.216058 (+ 118us) server_negotiation.cc:407] Connection header received
0924 15:27:14.216102 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:14.216103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:14.216107 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:14.216174 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:14.216744 (+ 570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:14.216745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:14.216745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:14.216750 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:14.216764 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:14.216769 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:14.216836 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:14.216893 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 15:27:14.216905 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:27:14.223951 25652 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:14.224114 25652 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:14.228546 25652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7b762b3c7fc4b05ab61d44ff0cd5614: Generated new TSK 0
I20260924 15:27:14.228746 25652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:14.235102 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25571
2026-09-24T15:27:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:14 dist-test-slave-l766 krb5kdc[25670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:14 dist-test-slave-l766 krb5kdc[25670](info): set up 2 sockets
Sep 24 15:27:14 dist-test-slave-l766 krb5kdc[25670](info): commencing operation
krb5kdc: starting...
W20260924 15:27:15.277088 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:15 dist-test-slave-l766 krb5kdc[25670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:15.392465 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36953
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:15.474067 25683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:15.474277 25683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:15.474299 25683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:15.475383 25683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:15.475417 25683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:15.475431 25683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:15.475443 25683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:15.475455 25683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:15.476838 25683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36953
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:15.477029 25683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:15.477190 25683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:15.481930 25688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:15.481925 25691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:15.481925 25689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:15.481941 25683 server_base.cc:1034] running on GCE node
I20260924 15:27:15.482457 25683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:15.482769 25683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:15.483933 25683 hybrid_clock.cc:648] HybridClock initialized: now 1790263635483921 us; error 44 us; skew 500 ppm
Sep 24 15:27:15 dist-test-slave-l766 krb5kdc[25670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263635, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:15.486775 25683 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:15.487206 25683 webserver.cc:460] Webserver started at http://127.0.0.1:35765/ using document root <none> and password file <none>
I20260924 15:27:15.487396 25683 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:15.487434 25683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:15.487515 25683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:15.488462 25683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "0f44681973de43a28cc8e0ff064f39e1"
format_stamp: "Formatted at 2026-09-24 15:27:15 on dist-test-slave-l766"
server_key: "dfdd9eaad14b2a1db4c04487a398b903"
server_key_iv: "0081791e110eb362445bd8ee3cac1bc1"
server_key_version: "encryptionkey@0"
I20260924 15:27:15.488749 25683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "0f44681973de43a28cc8e0ff064f39e1"
format_stamp: "Formatted at 2026-09-24 15:27:15 on dist-test-slave-l766"
server_key: "dfdd9eaad14b2a1db4c04487a398b903"
server_key_iv: "0081791e110eb362445bd8ee3cac1bc1"
server_key_version: "encryptionkey@0"
I20260924 15:27:15.490088 25683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:15.490756 25698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:15.490950 25683 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:15.491061 25683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "0f44681973de43a28cc8e0ff064f39e1"
format_stamp: "Formatted at 2026-09-24 15:27:15 on dist-test-slave-l766"
server_key: "dfdd9eaad14b2a1db4c04487a398b903"
server_key_iv: "0081791e110eb362445bd8ee3cac1bc1"
server_key_version: "encryptionkey@0"
I20260924 15:27:15.491128 25683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:15.503558 25683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:15.505569 25683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:15.505733 25683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:15.509475 25683 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36953
I20260924 15:27:15.509562 25750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36953 every 8 connection(s)
I20260924 15:27:15.509948 25683 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:15.510982 25751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:15.513247 25751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1: Bootstrap starting.
I20260924 15:27:15.513777 25751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:15.514025 25751 log.cc:824] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:15.514698 25751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1: No bootstrap required, opened a new log
I20260924 15:27:15.516237 25751 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f44681973de43a28cc8e0ff064f39e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260924 15:27:15.516383 25751 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:15.516407 25751 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f44681973de43a28cc8e0ff064f39e1, State: Initialized, Role: FOLLOWER
I20260924 15:27:15.516508 25751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f44681973de43a28cc8e0ff064f39e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260924 15:27:15.516593 25751 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:15.516630 25751 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:15.516705 25751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:15.516963 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25683
I20260924 15:27:15.517069 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:15.517259 4409 external_mini_cluster.cc:1428] Setting key f5f7b480fb6100379eea6ead89b29329
I20260924 15:27:15.517459 25751 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f44681973de43a28cc8e0ff064f39e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260924 15:27:15.517563 25751 leader_election.cc:302] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f44681973de43a28cc8e0ff064f39e1; no voters:
I20260924 15:27:15.517767 25751 leader_election.cc:288] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:15.517793 25754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:15.517925 25754 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [term 1 LEADER]: Becoming Leader. State: Replica: 0f44681973de43a28cc8e0ff064f39e1, State: Running, Role: LEADER
I20260924 15:27:15.518054 25751 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:15.518069 25754 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f44681973de43a28cc8e0ff064f39e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } }
I20260924 15:27:15.518409 25754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f44681973de43a28cc8e0ff064f39e1. Latest consensus state: current_term: 1 leader_uuid: "0f44681973de43a28cc8e0ff064f39e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f44681973de43a28cc8e0ff064f39e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } } }
I20260924 15:27:15.518493 25754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:15.518559 25755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f44681973de43a28cc8e0ff064f39e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f44681973de43a28cc8e0ff064f39e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36953 } } }
I20260924 15:27:15.518668 25755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:15.518735 25762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:15.519511 25762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:15.522969 25762 catalog_manager.cc:1347] Generated new cluster ID: 47c1aff7d4254e719520ab58e0fe07e5
I20260924 15:27:15.523046 25762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:15 dist-test-slave-l766 krb5kdc[25670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:27:15.529929 25762 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:15.529932 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:15.517992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:59243 (local address 127.0.0.1:36953)
0924 15:27:15.519412 (+ 1420us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:15.519422 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:15.520288 (+ 866us) server_negotiation.cc:407] Connection header received
0924 15:27:15.520420 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:15.520425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:15.520471 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:15.520581 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:15.521370 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:15.522393 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:15.523426 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:15.523875 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:15.524008 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:15.526450 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:15.526454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:15.526454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:15.526466 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:15.528151 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:15.528562 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:15.528569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:15.528570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:15.528663 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:15.528902 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:15.528903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:15.528904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:15.529139 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:15.529332 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:15.529421 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 15:27:15.529664 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1369,"thread_start_us":74,"threads_started":1}
I20260924 15:27:15.530603 25762 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:15.530939 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:15.538573 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:15.532283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:48773 (local address 127.0.0.1:36953)
0924 15:27:15.532320 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:15.532323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:15.532444 (+ 121us) server_negotiation.cc:407] Connection header received
0924 15:27:15.532502 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:15.532503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:15.532513 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:15.532567 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:15.533077 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:15.533601 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:15.534418 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:15.534640 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:15.534712 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:15.535951 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:15.535952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:15.535952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:15.535960 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:15.537381 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:15.537757 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:15.537758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:15.537758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:15.537831 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:15.538039 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:15.538040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:15.538040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:15.538148 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:15.538285 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:15.538394 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:27:15.538457 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:27:15.538628 25762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f44681973de43a28cc8e0ff064f39e1: Generated new TSK 0
I20260924 15:27:15.538795 25762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:15.546293 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25683
W20260924 15:27:15.550647 25668 connection.cc:535] client connection to 127.0.0.1:36953 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36953): Connection reset by peer (error 104)
2026-09-24T15:27:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:15 dist-test-slave-l766 krb5kdc[25782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:15 dist-test-slave-l766 krb5kdc[25782](info): set up 2 sockets
Sep 24 15:27:15 dist-test-slave-l766 krb5kdc[25782](info): commencing operation
krb5kdc: starting...
W20260924 15:27:16.577033 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:16 dist-test-slave-l766 krb5kdc[25782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263636, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:16.694766 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35261
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:16.775025 25795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:16.775265 25795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:16.775301 25795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:16.776430 25795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:16.776499 25795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:16.776525 25795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:16.776553 25795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:16.776584 25795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:16.778064 25795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35261
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:16.778335 25795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:16.778558 25795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:16.783411 25800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:16.783394 25801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:16.783425 25795 server_base.cc:1034] running on GCE node
W20260924 15:27:16.783390 25803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:16.783890 25795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:16.784183 25795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:16.785349 25795 hybrid_clock.cc:648] HybridClock initialized: now 1790263636785334 us; error 49 us; skew 500 ppm
Sep 24 15:27:16 dist-test-slave-l766 krb5kdc[25782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:16.793951 25795 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:16.794406 25795 webserver.cc:460] Webserver started at http://127.0.0.1:40709/ using document root <none> and password file <none>
I20260924 15:27:16.794646 25795 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:16.794723 25795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:16.794875 25795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:16.796000 25795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "0c01c3b528cf4a349e2588e7809b5044"
format_stamp: "Formatted at 2026-09-24 15:27:16 on dist-test-slave-l766"
server_key: "e0f8e03245185fbbeabe52e599f5d9d7"
server_key_iv: "8512106f666346f0933b3850a893719a"
server_key_version: "encryptionkey@0"
I20260924 15:27:16.796361 25795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "0c01c3b528cf4a349e2588e7809b5044"
format_stamp: "Formatted at 2026-09-24 15:27:16 on dist-test-slave-l766"
server_key: "e0f8e03245185fbbeabe52e599f5d9d7"
server_key_iv: "8512106f666346f0933b3850a893719a"
server_key_version: "encryptionkey@0"
I20260924 15:27:16.797799 25795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:16.798668 25810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:16.798871 25795 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:16.798980 25795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "0c01c3b528cf4a349e2588e7809b5044"
format_stamp: "Formatted at 2026-09-24 15:27:16 on dist-test-slave-l766"
server_key: "e0f8e03245185fbbeabe52e599f5d9d7"
server_key_iv: "8512106f666346f0933b3850a893719a"
server_key_version: "encryptionkey@0"
I20260924 15:27:16.799060 25795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:16.844300 25795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:16.846801 25795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:16.846925 25795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:16.851082 25795 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33815
I20260924 15:27:16.851140 25862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33815 every 8 connection(s)
I20260924 15:27:16.851486 25795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:16.852428 25863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:16.854686 25863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044: Bootstrap starting.
I20260924 15:27:16.855214 25863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:16.855445 25863 log.cc:824] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:16.856144 25863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044: No bootstrap required, opened a new log
I20260924 15:27:16.857702 25863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01c3b528cf4a349e2588e7809b5044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33815 } }
I20260924 15:27:16.857929 25863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:16.857962 25863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c01c3b528cf4a349e2588e7809b5044, State: Initialized, Role: FOLLOWER
I20260924 15:27:16.858074 25863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01c3b528cf4a349e2588e7809b5044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33815 } }
I20260924 15:27:16.858129 25863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:16.858148 25863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:16.858174 25863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:16.858659 25863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01c3b528cf4a349e2588e7809b5044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33815 } }
I20260924 15:27:16.858747 25863 leader_election.cc:302] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c01c3b528cf4a349e2588e7809b5044; no voters:
I20260924 15:27:16.858870 25863 leader_election.cc:288] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:16.858917 25866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:16.859087 25866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [term 1 LEADER]: Becoming Leader. State: Replica: 0c01c3b528cf4a349e2588e7809b5044, State: Running, Role: LEADER
I20260924 15:27:16.859112 25863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:16.859215 25866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01c3b528cf4a349e2588e7809b5044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33815 } }
I20260924 15:27:16.859658 25868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c01c3b528cf4a349e2588e7809b5044. Latest consensus state: current_term: 1 leader_uuid: "0c01c3b528cf4a349e2588e7809b5044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01c3b528cf4a349e2588e7809b5044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33815 } } }
I20260924 15:27:16.859768 25868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:16.859844 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25795
I20260924 15:27:16.859977 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:16.860208 4409 external_mini_cluster.cc:1428] Setting key cad2ca186f327591c09478cfb3dff3fd
I20260924 15:27:16.860425 25867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c01c3b528cf4a349e2588e7809b5044" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c01c3b528cf4a349e2588e7809b5044" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33815 } } }
I20260924 15:27:16.860910 25867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044 [sys.catalog]: This master's current role is: LEADER
W20260924 15:27:16.861037 25882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:16.861217 25882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:16.861289 25875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:16.861461 25875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:16.863026 25875 catalog_manager.cc:1347] Generated new cluster ID: 96bbb48c4af54ae685c12330d6dd6727
I20260924 15:27:16.863080 25875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:16.866604 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:16.860721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:49777 (local address 127.0.0.1:33815)
0924 15:27:16.860991 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:16.861000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:16.861036 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:27:16.861160 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:16.861170 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:16.861243 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:16.862013 (+ 770us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:16.862246 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:16.863363 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:16.864469 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:16.865071 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:16.865164 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:16.865909 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:16.865913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:16.865913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:16.866127 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:16.866162 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:16.866177 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:16.866256 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:16.866365 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:27:16.866392 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260924 15:27:16.867686 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:16.869781 25875 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:16.870544 25875 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:16.872201 25885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:16.869212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:51571 (local address 127.0.0.1:33815)
0924 15:27:16.869247 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:16.869249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:16.869345 (+ 96us) server_negotiation.cc:407] Connection header received
0924 15:27:16.869392 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:16.869393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:16.869403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:16.869464 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:16.869574 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:16.870152 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:16.870975 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:16.871173 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:16.871251 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:16.871927 (+ 676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:16.871928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:16.871928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:16.871933 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:16.871948 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:16.871953 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:16.872028 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:16.872093 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:27:16.872104 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260924 15:27:16.874233 25875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c01c3b528cf4a349e2588e7809b5044: Generated new TSK 0
I20260924 15:27:16.874405 25875 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:16.876835 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25795
W20260924 15:27:16.881282 25780 connection.cc:535] client connection to 127.0.0.1:33815 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33815): Connection reset by peer (error 104)
2026-09-24T15:27:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:16 dist-test-slave-l766 krb5kdc[25894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:16 dist-test-slave-l766 krb5kdc[25894](info): set up 2 sockets
Sep 24 15:27:16 dist-test-slave-l766 krb5kdc[25894](info): commencing operation
krb5kdc: starting...
W20260924 15:27:17.909005 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:17 dist-test-slave-l766 krb5kdc[25894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263637, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:27:18.029103 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43767
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:18.109510 25907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:18.109720 25907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:18.109745 25907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:18.110934 25907 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:18.110975 25907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:18.110989 25907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:18.111002 25907 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:18.111014 25907 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:18.112480 25907 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43767
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:18.112732 25907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:18.112957 25907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:18.117815 25912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:18.117846 25915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:18.117847 25913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:18.118054 25907 server_base.cc:1034] running on GCE node
I20260924 15:27:18.118310 25907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:18.118580 25907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:18.119760 25907 hybrid_clock.cc:648] HybridClock initialized: now 1790263638119736 us; error 37 us; skew 500 ppm
Sep 24 15:27:18 dist-test-slave-l766 krb5kdc[25894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263638, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:18.122097 25907 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:27:18.122568 25907 webserver.cc:460] Webserver started at http://127.0.0.1:37773/ using document root <none> and password file <none>
I20260924 15:27:18.122787 25907 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:18.122852 25907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:18.123004 25907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:18.124027 25907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "a23f0fa71c5943dd93fab5f806bb854f"
format_stamp: "Formatted at 2026-09-24 15:27:18 on dist-test-slave-l766"
server_key: "001930b0e9e9b98631f02dd0a953934c"
server_key_iv: "6e24f2e34866d3b32823bef6c8d92d62"
server_key_version: "encryptionkey@0"
I20260924 15:27:18.124380 25907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "a23f0fa71c5943dd93fab5f806bb854f"
format_stamp: "Formatted at 2026-09-24 15:27:18 on dist-test-slave-l766"
server_key: "001930b0e9e9b98631f02dd0a953934c"
server_key_iv: "6e24f2e34866d3b32823bef6c8d92d62"
server_key_version: "encryptionkey@0"
I20260924 15:27:18.125674 25907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:18.126485 25922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:18.126668 25907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:18.126765 25907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "a23f0fa71c5943dd93fab5f806bb854f"
format_stamp: "Formatted at 2026-09-24 15:27:18 on dist-test-slave-l766"
server_key: "001930b0e9e9b98631f02dd0a953934c"
server_key_iv: "6e24f2e34866d3b32823bef6c8d92d62"
server_key_version: "encryptionkey@0"
I20260924 15:27:18.126849 25907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:18.130067 25907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:18.131819 25907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:18.131970 25907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:18.135861 25907 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43767
I20260924 15:27:18.135907 25975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43767 every 8 connection(s)
I20260924 15:27:18.136216 25907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:18.137135 25976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:18.139456 25976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f: Bootstrap starting.
I20260924 15:27:18.140012 25976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:18.140293 25976 log.cc:824] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:18.140947 25976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f: No bootstrap required, opened a new log
I20260924 15:27:18.142623 25976 raft_consensus.cc:348] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23f0fa71c5943dd93fab5f806bb854f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43767 } }
I20260924 15:27:18.142761 25976 raft_consensus.cc:374] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:18.142807 25976 raft_consensus.cc:729] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a23f0fa71c5943dd93fab5f806bb854f, State: Initialized, Role: FOLLOWER
I20260924 15:27:18.142973 25976 consensus_queue.cc:260] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23f0fa71c5943dd93fab5f806bb854f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43767 } }
I20260924 15:27:18.143033 25976 raft_consensus.cc:388] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:18.143072 25976 raft_consensus.cc:482] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:18.143121 25976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:18.143657 25976 raft_consensus.cc:504] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23f0fa71c5943dd93fab5f806bb854f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43767 } }
I20260924 15:27:18.143766 25976 leader_election.cc:302] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a23f0fa71c5943dd93fab5f806bb854f; no voters:
I20260924 15:27:18.143951 25976 leader_election.cc:288] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:18.143983 25979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:18.144088 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 25907
I20260924 15:27:18.144176 25979 raft_consensus.cc:686] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [term 1 LEADER]: Becoming Leader. State: Replica: a23f0fa71c5943dd93fab5f806bb854f, State: Running, Role: LEADER
I20260924 15:27:18.144193 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:18.144234 25976 sys_catalog.cc:564] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:18.144279 25979 consensus_queue.cc:237] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23f0fa71c5943dd93fab5f806bb854f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43767 } }
I20260924 15:27:18.144375 4409 external_mini_cluster.cc:1428] Setting key 2a331a9ac3c393ac1bda07fa8379b966
I20260924 15:27:18.144588 25979 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a23f0fa71c5943dd93fab5f806bb854f. Latest consensus state: current_term: 1 leader_uuid: "a23f0fa71c5943dd93fab5f806bb854f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23f0fa71c5943dd93fab5f806bb854f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43767 } } }
I20260924 15:27:18.144660 25979 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:18.144747 25980 sys_catalog.cc:455] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a23f0fa71c5943dd93fab5f806bb854f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a23f0fa71c5943dd93fab5f806bb854f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43767 } } }
I20260924 15:27:18.144802 25980 sys_catalog.cc:458] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:18.144862 25985 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:18.145390 25985 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:18.147089 25985 catalog_manager.cc:1347] Generated new cluster ID: a5d42dd3af0f4d3ca0717ebe30d89b91
I20260924 15:27:18.147143 25985 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:18.155504 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:18.145026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:54789 (local address 127.0.0.1:43767)
0924 15:27:18.147007 (+ 1981us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:18.147016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:18.153964 (+ 6948us) server_negotiation.cc:407] Connection header received
0924 15:27:18.154075 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:18.154080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:18.154113 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:18.154272 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:18.155012 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:18.155016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:18.155016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:18.155020 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:18.155069 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:18.155080 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:18.155155 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:18.155230 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:27:18.155321 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1931,"thread_start_us":74,"threads_started":1}
I20260924 15:27:18.156507 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:18.157068 25985 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:18.157223 25985 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:18.159210 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:18.158097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.231:37805 (local address 127.0.0.1:43767)
0924 15:27:18.158131 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:18.158132 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:18.158329 (+ 197us) server_negotiation.cc:407] Connection header received
0924 15:27:18.158356 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:18.158357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:18.158362 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:18.158430 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:18.159005 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:18.159006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:18.159006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:18.159010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:18.159025 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:18.159029 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:18.159087 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:18.159137 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 15:27:18.159149 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:27:18.168497 25985 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a23f0fa71c5943dd93fab5f806bb854f: Generated new TSK 0
I20260924 15:27:18.168709 25985 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:18.170130 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 25907
2026-09-24T15:27:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:18 dist-test-slave-l766 krb5kdc[26007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:18 dist-test-slave-l766 krb5kdc[26007](info): set up 2 sockets
Sep 24 15:27:18 dist-test-slave-l766 krb5kdc[26007](info): commencing operation
krb5kdc: starting...
W20260924 15:27:19.261957 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:19 dist-test-slave-l766 krb5kdc[26007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:19.380190 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:41783
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:39867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:41783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:19.460494 26020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:19.460690 26020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:19.460713 26020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:19.461771 26020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:19.461825 26020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:19.461841 26020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:19.461892 26020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:19.461922 26020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:19.463366 26020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:39867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:41783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:41783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:19.463606 26020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:19.463821 26020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:19.468700 26028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:19.468700 26025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:19.468863 26020 server_base.cc:1034] running on GCE node
W20260924 15:27:19.468866 26026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:19.469218 26020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:19.469519 26020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:19.470721 26020 hybrid_clock.cc:648] HybridClock initialized: now 1790263639470696 us; error 40 us; skew 500 ppm
Sep 24 15:27:19 dist-test-slave-l766 krb5kdc[26007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263639, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:19.473129 26020 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:19.473560 26020 webserver.cc:460] Webserver started at http://127.4.78.126:33765/ using document root <none> and password file <none>
I20260924 15:27:19.473850 26020 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:19.473933 26020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:19.474098 26020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:19.475288 26020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e0980aaeb3264be8997fc2ea8f6e06de"
format_stamp: "Formatted at 2026-09-24 15:27:19 on dist-test-slave-l766"
server_key: "da295a8a42ca3f5ad570c7e7d14ea62b"
server_key_iv: "19dea0e9d8278c1680c7bfc25567cc42"
server_key_version: "encryptionkey@0"
I20260924 15:27:19.475649 26020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e0980aaeb3264be8997fc2ea8f6e06de"
format_stamp: "Formatted at 2026-09-24 15:27:19 on dist-test-slave-l766"
server_key: "da295a8a42ca3f5ad570c7e7d14ea62b"
server_key_iv: "19dea0e9d8278c1680c7bfc25567cc42"
server_key_version: "encryptionkey@0"
I20260924 15:27:19.477185 26020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:19.477954 26035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:19.478129 26020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:19.478262 26020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e0980aaeb3264be8997fc2ea8f6e06de"
format_stamp: "Formatted at 2026-09-24 15:27:19 on dist-test-slave-l766"
server_key: "da295a8a42ca3f5ad570c7e7d14ea62b"
server_key_iv: "19dea0e9d8278c1680c7bfc25567cc42"
server_key_version: "encryptionkey@0"
I20260924 15:27:19.478343 26020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:19.492708 26020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:19.494611 26020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:19.494777 26020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:19.498631 26020 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:41783
I20260924 15:27:19.498685 26087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:41783 every 8 connection(s)
I20260924 15:27:19.499079 26020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:19.499951 26088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:19.502139 26088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de: Bootstrap starting.
I20260924 15:27:19.502689 26088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:19.502969 26088 log.cc:824] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:19.503643 26088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de: No bootstrap required, opened a new log
I20260924 15:27:19.504887 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26020
I20260924 15:27:19.505003 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:19.505183 4409 external_mini_cluster.cc:1428] Setting key f00370a068e01570ff5aedcdfb648c01
I20260924 15:27:19.505545 26088 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0980aaeb3264be8997fc2ea8f6e06de" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41783 } }
I20260924 15:27:19.505741 26088 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:19.505867 26088 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0980aaeb3264be8997fc2ea8f6e06de, State: Initialized, Role: FOLLOWER
I20260924 15:27:19.506042 26088 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0980aaeb3264be8997fc2ea8f6e06de" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41783 } }
I20260924 15:27:19.506137 26088 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:19.506183 26088 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:19.506232 26088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:19.507104 26088 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0980aaeb3264be8997fc2ea8f6e06de" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41783 } }
I20260924 15:27:19.507206 26088 leader_election.cc:302] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0980aaeb3264be8997fc2ea8f6e06de; no voters:
I20260924 15:27:19.507349 26088 leader_election.cc:288] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:19.507421 26093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:19.507580 26088 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:19.507589 26093 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [term 1 LEADER]: Becoming Leader. State: Replica: e0980aaeb3264be8997fc2ea8f6e06de, State: Running, Role: LEADER
I20260924 15:27:19.507761 26093 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0980aaeb3264be8997fc2ea8f6e06de" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41783 } }
I20260924 15:27:19.508164 26095 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0980aaeb3264be8997fc2ea8f6e06de. Latest consensus state: current_term: 1 leader_uuid: "e0980aaeb3264be8997fc2ea8f6e06de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0980aaeb3264be8997fc2ea8f6e06de" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41783 } } }
I20260924 15:27:19.508293 26095 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:19.508507 26102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:19.508930 26094 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0980aaeb3264be8997fc2ea8f6e06de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0980aaeb3264be8997fc2ea8f6e06de" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 41783 } } }
I20260924 15:27:19.509258 26094 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:19.509023 26102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:19.511276 26102 catalog_manager.cc:1347] Generated new cluster ID: d700043d734e4e66b641eae08b57b522
I20260924 15:27:19.511337 26102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:19 dist-test-slave-l766 krb5kdc[26007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263639, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:19.517216 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:19.505695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58491 (local address 127.4.78.126:41783)
0924 15:27:19.505903 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:19.505914 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:19.505944 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:27:19.506076 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:19.506081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:19.506125 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:19.506277 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:19.507001 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:19.508452 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:19.509626 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:19.510114 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:19.510218 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:19.512990 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:19.512994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:19.512994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:19.513005 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:19.515029 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:19.515522 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:19.515525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:19.515526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:19.515645 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:19.515902 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:19.515903 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:19.515903 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:19.516214 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:19.516382 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:19.516825 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 15:27:19.516948 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":88,"threads_started":1}
I20260924 15:27:19.518293 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:19.526705 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:19.519673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50899 (local address 127.4.78.126:41783)
0924 15:27:19.519711 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:19.519713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:19.519846 (+ 133us) server_negotiation.cc:407] Connection header received
0924 15:27:19.519875 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:19.519876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:19.519888 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:19.519955 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:19.520516 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:19.521256 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:19.522224 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:19.522552 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:19.522632 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:19.523919 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:19.523921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:19.523921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:19.523928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:19.525480 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:19.525893 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:19.525894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:19.525894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:19.525960 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:19.526188 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:19.526189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:19.526189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:19.526313 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:19.526411 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:19.526551 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 15:27:19.526614 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:27:19.534905 26102 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:19.535629 26102 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:19.548866 26102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0980aaeb3264be8997fc2ea8f6e06de: Generated new TSK 0
I20260924 15:27:19.549072 26102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:19.559330 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26020
W20260924 15:27:19.563361 26005 connection.cc:535] client connection to 127.4.78.126:41783 recv error: Network error: failed to read from TLS socket (remote: 127.4.78.126:41783): Connection reset by peer (error 104)
2026-09-24T15:27:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:19 dist-test-slave-l766 krb5kdc[26119](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:19 dist-test-slave-l766 krb5kdc[26119](info): set up 2 sockets
Sep 24 15:27:19 dist-test-slave-l766 krb5kdc[26119](info): commencing operation
krb5kdc: starting...
W20260924 15:27:20.589015 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:20 dist-test-slave-l766 krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:20.703155 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:44743
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:44959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:44743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:20.783212 26132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:20.783422 26132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:20.783445 26132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:20.784600 26132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:20.784642 26132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:20.784657 26132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:20.784669 26132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:20.784723 26132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:20.786185 26132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:44959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:44743
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:44743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:20.786432 26132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:20.786695 26132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:20.791545 26140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:20.791561 26138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:20.791545 26137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:20.791823 26132 server_base.cc:1034] running on GCE node
I20260924 15:27:20.792049 26132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:20.792382 26132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:20.793560 26132 hybrid_clock.cc:648] HybridClock initialized: now 1790263640793543 us; error 47 us; skew 500 ppm
Sep 24 15:27:20 dist-test-slave-l766 krb5kdc[26119](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263640, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:20.795970 26132 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:20.796334 26132 webserver.cc:460] Webserver started at http://127.4.78.126:35205/ using document root <none> and password file <none>
I20260924 15:27:20.796522 26132 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:20.796562 26132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:20.796646 26132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:20.797688 26132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "09b353f02b8e4d5991268013d8fa1b2a"
format_stamp: "Formatted at 2026-09-24 15:27:20 on dist-test-slave-l766"
server_key: "9a9504d32fd70b3f90ef4fba3ce3abb1"
server_key_iv: "c806cf17250c520eee4595d177eaf147"
server_key_version: "encryptionkey@0"
I20260924 15:27:20.798035 26132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "09b353f02b8e4d5991268013d8fa1b2a"
format_stamp: "Formatted at 2026-09-24 15:27:20 on dist-test-slave-l766"
server_key: "9a9504d32fd70b3f90ef4fba3ce3abb1"
server_key_iv: "c806cf17250c520eee4595d177eaf147"
server_key_version: "encryptionkey@0"
I20260924 15:27:20.799317 26132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:27:20.799947 26147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:20.800117 26132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:20.800220 26132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "09b353f02b8e4d5991268013d8fa1b2a"
format_stamp: "Formatted at 2026-09-24 15:27:20 on dist-test-slave-l766"
server_key: "9a9504d32fd70b3f90ef4fba3ce3abb1"
server_key_iv: "c806cf17250c520eee4595d177eaf147"
server_key_version: "encryptionkey@0"
I20260924 15:27:20.800282 26132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:20.818977 26132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:20.820966 26132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:20.821125 26132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:20.824970 26132 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:44743
I20260924 15:27:20.825035 26199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:44743 every 8 connection(s)
I20260924 15:27:20.825376 26132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:20.826377 26200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:20.827937 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26132
I20260924 15:27:20.828058 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:20.828259 4409 external_mini_cluster.cc:1428] Setting key b0bf2ef905fd2115bac5659016c9819b
I20260924 15:27:20.829064 26200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a: Bootstrap starting.
I20260924 15:27:20.829697 26200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:20.830039 26200 log.cc:824] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:20.830731 26200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a: No bootstrap required, opened a new log
I20260924 15:27:20.833143 26200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b353f02b8e4d5991268013d8fa1b2a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44743 } }
I20260924 15:27:20.833323 26200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:20.833369 26200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b353f02b8e4d5991268013d8fa1b2a, State: Initialized, Role: FOLLOWER
I20260924 15:27:20.833498 26200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b353f02b8e4d5991268013d8fa1b2a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44743 } }
I20260924 15:27:20.833599 26200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:20.833639 26200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:20.833719 26200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:20.833976 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:20.828774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42301 (local address 127.4.78.126:44743)
0924 15:27:20.828954 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:20.828963 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:20.828988 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:27:20.829455 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:20.829463 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:20.829510 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:20.829624 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:20.829902 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:20.830890 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:20.831981 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:20.832373 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:20.832460 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:20.833198 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:20.833201 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:20.833201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:20.833403 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:20.833442 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:20.833457 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:20.833549 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:20.833681 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 15:27:20.833712 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260924 15:27:20.834465 26200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b353f02b8e4d5991268013d8fa1b2a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44743 } }
I20260924 15:27:20.834575 26200 leader_election.cc:302] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09b353f02b8e4d5991268013d8fa1b2a; no voters:
I20260924 15:27:20.834743 26200 leader_election.cc:288] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:20.834789 26205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:20.834932 26205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [term 1 LEADER]: Becoming Leader. State: Replica: 09b353f02b8e4d5991268013d8fa1b2a, State: Running, Role: LEADER
I20260924 15:27:20.834997 26200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:20.835021 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:20.835068 26205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b353f02b8e4d5991268013d8fa1b2a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44743 } }
I20260924 15:27:20.835467 26206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09b353f02b8e4d5991268013d8fa1b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b353f02b8e4d5991268013d8fa1b2a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44743 } } }
I20260924 15:27:20.835546 26206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:20.835740 26207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09b353f02b8e4d5991268013d8fa1b2a. Latest consensus state: current_term: 1 leader_uuid: "09b353f02b8e4d5991268013d8fa1b2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b353f02b8e4d5991268013d8fa1b2a" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44743 } } }
I20260924 15:27:20.835855 26207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:20.836637 26212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:20.837136 26212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:20.838716 26212 catalog_manager.cc:1347] Generated new cluster ID: 1da8e32e8485427d98793a9609daee42
I20260924 15:27:20.838769 26212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:20.839844 26203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:20.836573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32885 (local address 127.4.78.126:44743)
0924 15:27:20.836631 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:20.836633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:20.836785 (+ 152us) server_negotiation.cc:407] Connection header received
0924 15:27:20.836856 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:20.836857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:20.836867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:20.836916 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:20.837033 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:20.837578 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:20.838569 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:20.838866 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:20.838931 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:20.839613 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:20.839614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:20.839614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:20.839618 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:20.839629 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:20.839633 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:20.839660 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:20.839768 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 15:27:20.839777 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:20.857910 26212 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:20.858485 26212 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:20.861948 26212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09b353f02b8e4d5991268013d8fa1b2a: Generated new TSK 0
I20260924 15:27:20.862133 26212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:20.864261 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26132
W20260924 15:27:20.868481 26117 connection.cc:535] client connection to 127.4.78.126:44743 recv error: Network error: failed to read from TLS socket (remote: 127.4.78.126:44743): Connection reset by peer (error 104)
2026-09-24T15:27:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:20 dist-test-slave-l766 krb5kdc[26231](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:20 dist-test-slave-l766 krb5kdc[26231](info): set up 2 sockets
Sep 24 15:27:20 dist-test-slave-l766 krb5kdc[26231](info): commencing operation
krb5kdc: starting...
W20260924 15:27:21.908977 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:21 dist-test-slave-l766 krb5kdc[26231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263641, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:22.028419 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42861
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:42861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:22.109580 26244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:22.109824 26244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:22.109903 26244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:22.111093 26244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:22.111166 26244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:22.111191 26244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:22.111219 26244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:22.111251 26244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:22.112814 26244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:42861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:22.113085 26244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:22.113324 26244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:22.118132 26252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:22.118283 26244 server_base.cc:1034] running on GCE node
W20260924 15:27:22.118285 26250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:22.118140 26249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:22.118673 26244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:22.118994 26244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:22.120160 26244 hybrid_clock.cc:648] HybridClock initialized: now 1790263642120147 us; error 27 us; skew 500 ppm
Sep 24 15:27:22 dist-test-slave-l766 krb5kdc[26231](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263642, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:22.130041 26244 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:22.130494 26244 webserver.cc:460] Webserver started at http://127.4.78.126:40121/ using document root <none> and password file <none>
I20260924 15:27:22.130741 26244 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:22.130811 26244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:22.130959 26244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:22.132025 26244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e812644c6f324e63a3e024352feab581"
format_stamp: "Formatted at 2026-09-24 15:27:22 on dist-test-slave-l766"
server_key: "afd392eaa8e2b8d6c4ee23eae5c830de"
server_key_iv: "92fe0a5daec022c712ff25d9eee7735b"
server_key_version: "encryptionkey@0"
I20260924 15:27:22.132378 26244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e812644c6f324e63a3e024352feab581"
format_stamp: "Formatted at 2026-09-24 15:27:22 on dist-test-slave-l766"
server_key: "afd392eaa8e2b8d6c4ee23eae5c830de"
server_key_iv: "92fe0a5daec022c712ff25d9eee7735b"
server_key_version: "encryptionkey@0"
I20260924 15:27:22.133703 26244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:22.134495 26259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:22.134665 26244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:22.134764 26244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e812644c6f324e63a3e024352feab581"
format_stamp: "Formatted at 2026-09-24 15:27:22 on dist-test-slave-l766"
server_key: "afd392eaa8e2b8d6c4ee23eae5c830de"
server_key_iv: "92fe0a5daec022c712ff25d9eee7735b"
server_key_version: "encryptionkey@0"
I20260924 15:27:22.134846 26244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:22.138095 26244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:22.139838 26244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:22.139999 26244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:22.143883 26244 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:42861
I20260924 15:27:22.143931 26311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:42861 every 8 connection(s)
I20260924 15:27:22.144236 26244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:22.145161 26312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:22.147428 26312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581: Bootstrap starting.
I20260924 15:27:22.147997 26312 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:22.148283 26312 log.cc:824] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:22.148937 26312 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581: No bootstrap required, opened a new log
I20260924 15:27:22.150575 26312 raft_consensus.cc:348] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e812644c6f324e63a3e024352feab581" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42861 } }
I20260924 15:27:22.150707 26312 raft_consensus.cc:374] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:22.150766 26312 raft_consensus.cc:729] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e812644c6f324e63a3e024352feab581, State: Initialized, Role: FOLLOWER
I20260924 15:27:22.150934 26312 consensus_queue.cc:260] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e812644c6f324e63a3e024352feab581" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42861 } }
I20260924 15:27:22.150998 26312 raft_consensus.cc:388] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:22.151037 26312 raft_consensus.cc:482] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:22.151086 26312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:22.151908 26312 raft_consensus.cc:504] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e812644c6f324e63a3e024352feab581" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42861 } }
I20260924 15:27:22.152021 26312 leader_election.cc:302] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e812644c6f324e63a3e024352feab581; no voters:
I20260924 15:27:22.152221 26312 leader_election.cc:288] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:22.152278 26315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:22.152472 26312 sys_catalog.cc:564] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:22.152485 26315 raft_consensus.cc:686] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [term 1 LEADER]: Becoming Leader. State: Replica: e812644c6f324e63a3e024352feab581, State: Running, Role: LEADER
I20260924 15:27:22.152606 26315 consensus_queue.cc:237] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e812644c6f324e63a3e024352feab581" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42861 } }
I20260924 15:27:22.153031 26316 sys_catalog.cc:455] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e812644c6f324e63a3e024352feab581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e812644c6f324e63a3e024352feab581" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42861 } } }
I20260924 15:27:22.153041 26317 sys_catalog.cc:455] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e812644c6f324e63a3e024352feab581. Latest consensus state: current_term: 1 leader_uuid: "e812644c6f324e63a3e024352feab581" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e812644c6f324e63a3e024352feab581" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42861 } } }
I20260924 15:27:22.153137 26316 sys_catalog.cc:458] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:22.153143 26317 sys_catalog.cc:458] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:22.153345 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26244
I20260924 15:27:22.153352 26324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:22.153451 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:22.153638 4409 external_mini_cluster.cc:1428] Setting key 85f9b8c082c892fceec409c0cfe21af4
I20260924 15:27:22.153880 26324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:22.156100 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:22.154249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36735 (local address 127.4.78.126:42861)
0924 15:27:22.154419 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:22.154428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:22.154659 (+ 231us) server_negotiation.cc:407] Connection header received
0924 15:27:22.154779 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:22.154786 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:22.154815 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:22.154965 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:22.155656 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:22.155660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:22.155660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:22.155665 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:22.155714 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:22.155729 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:22.155806 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:22.155893 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 15:27:22.155936 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260924 15:27:22.156996 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:22.158478 26324 catalog_manager.cc:1347] Generated new cluster ID: 5dd194dcf2264f6f9be6f43ad6d28dbb
I20260924 15:27:22.158529 26324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:22.159519 26333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:22.158384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40587 (local address 127.4.78.126:42861)
0924 15:27:22.158448 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:22.158450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:22.158581 (+ 131us) server_negotiation.cc:407] Connection header received
0924 15:27:22.158666 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:22.158667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:22.158671 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:22.158715 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:22.159301 (+ 586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:22.159302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:22.159303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:22.159306 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:22.159320 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:22.159324 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:22.159370 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:22.159424 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 15:27:22.159439 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:27:22.182463 26324 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:22.182701 26324 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:22.185557 26324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e812644c6f324e63a3e024352feab581: Generated new TSK 0
I20260924 15:27:22.185751 26324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:22.195562 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26244
2026-09-24T15:27:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:22 dist-test-slave-l766 krb5kdc[26343](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:22 dist-test-slave-l766 krb5kdc[26343](info): set up 2 sockets
Sep 24 15:27:22 dist-test-slave-l766 krb5kdc[26343](info): commencing operation
krb5kdc: starting...
W20260924 15:27:23.240936 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:23 dist-test-slave-l766 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:23.360320 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:34073
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:35157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:34073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:23.440804 26356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:23.441045 26356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:23.441082 26356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:23.442230 26356 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:23.442302 26356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:23.442327 26356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:23.442355 26356 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:23.442395 26356 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:23.443825 26356 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:35157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:34073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:34073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:23.444094 26356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:23.444314 26356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:23.449136 26364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:23.449128 26362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:23.449251 26361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:23.449170 26356 server_base.cc:1034] running on GCE node
I20260924 15:27:23.449601 26356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:23.449949 26356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:23.451105 26356 hybrid_clock.cc:648] HybridClock initialized: now 1790263643451083 us; error 43 us; skew 500 ppm
Sep 24 15:27:23 dist-test-slave-l766 krb5kdc[26343](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263643, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:23.453507 26356 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:23.453958 26356 webserver.cc:460] Webserver started at http://127.4.78.126:34281/ using document root <none> and password file <none>
I20260924 15:27:23.454137 26356 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:23.454180 26356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:23.454293 26356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:23.455322 26356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "bbd88d3e96004be5b2df23294018a955"
format_stamp: "Formatted at 2026-09-24 15:27:23 on dist-test-slave-l766"
server_key: "cb17326cf36bb9d6abea3d6758609cba"
server_key_iv: "2cceebd24600a93e06ca3aea60a98841"
server_key_version: "encryptionkey@0"
I20260924 15:27:23.455623 26356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "bbd88d3e96004be5b2df23294018a955"
format_stamp: "Formatted at 2026-09-24 15:27:23 on dist-test-slave-l766"
server_key: "cb17326cf36bb9d6abea3d6758609cba"
server_key_iv: "2cceebd24600a93e06ca3aea60a98841"
server_key_version: "encryptionkey@0"
I20260924 15:27:23.456858 26356 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:27:23.457502 26371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:23.457690 26356 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:23.457782 26356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "bbd88d3e96004be5b2df23294018a955"
format_stamp: "Formatted at 2026-09-24 15:27:23 on dist-test-slave-l766"
server_key: "cb17326cf36bb9d6abea3d6758609cba"
server_key_iv: "2cceebd24600a93e06ca3aea60a98841"
server_key_version: "encryptionkey@0"
I20260924 15:27:23.457877 26356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:23.500550 26356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:23.503010 26356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:23.503136 26356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:23.507058 26356 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:34073
I20260924 15:27:23.507138 26423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:34073 every 8 connection(s)
I20260924 15:27:23.507479 26356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:23.508374 26424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:23.510699 26424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955: Bootstrap starting.
I20260924 15:27:23.511270 26424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:23.511562 26424 log.cc:824] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:23.512277 26424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955: No bootstrap required, opened a new log
I20260924 15:27:23.513856 26424 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd88d3e96004be5b2df23294018a955" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34073 } }
I20260924 15:27:23.514029 26424 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:23.514077 26424 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd88d3e96004be5b2df23294018a955, State: Initialized, Role: FOLLOWER
I20260924 15:27:23.514267 26424 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd88d3e96004be5b2df23294018a955" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34073 } }
I20260924 15:27:23.514353 26424 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:23.514395 26424 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:23.514448 26424 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:23.515252 26424 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd88d3e96004be5b2df23294018a955" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34073 } }
I20260924 15:27:23.515372 26424 leader_election.cc:302] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbd88d3e96004be5b2df23294018a955; no voters:
I20260924 15:27:23.515548 26424 leader_election.cc:288] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:23.515578 26427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:23.515566 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26356
I20260924 15:27:23.515688 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:23.515693 26427 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [term 1 LEADER]: Becoming Leader. State: Replica: bbd88d3e96004be5b2df23294018a955, State: Running, Role: LEADER
I20260924 15:27:23.515769 26427 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd88d3e96004be5b2df23294018a955" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34073 } }
I20260924 15:27:23.515801 26424 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:23.515890 4409 external_mini_cluster.cc:1428] Setting key e13d1846d94193fc81c0174d724ab690
I20260924 15:27:23.516098 26429 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbd88d3e96004be5b2df23294018a955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd88d3e96004be5b2df23294018a955" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34073 } } }
I20260924 15:27:23.516184 26429 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:23.516378 26428 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd88d3e96004be5b2df23294018a955. Latest consensus state: current_term: 1 leader_uuid: "bbd88d3e96004be5b2df23294018a955" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd88d3e96004be5b2df23294018a955" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 34073 } } }
I20260924 15:27:23.516435 26428 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:23.516812 26433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:23.517434 26433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:23.519215 26433 catalog_manager.cc:1347] Generated new cluster ID: 0966a2f7ed7f46c3b021e71a654963bf
I20260924 15:27:23.519277 26433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:27:23 dist-test-slave-l766 krb5kdc[26343](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263643, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:23.527832 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:23.516592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49263 (local address 127.4.78.126:34073)
0924 15:27:23.516802 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:23.516813 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:23.516856 (+ 43us) server_negotiation.cc:407] Connection header received
0924 15:27:23.517000 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:23.517007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:23.517072 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:23.517235 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:23.518077 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:23.519436 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:23.520553 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:23.521183 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:23.521292 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:23.524141 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:23.524146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:23.524146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:23.524158 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:23.526166 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:23.526559 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:23.526562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:23.526562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:23.526680 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:23.526896 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:23.526897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:23.526897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:23.527193 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:23.527352 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:23.527467 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 15:27:23.527599 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":67,"threads_started":1}
I20260924 15:27:23.528839 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:23.530202 26433 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:23.530658 26433 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:23.536491 26433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbd88d3e96004be5b2df23294018a955: Generated new TSK 0
I20260924 15:27:23.536655 26433 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:23.541038 26439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:23.534096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45821 (local address 127.4.78.126:34073)
0924 15:27:23.534147 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:23.534149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:23.534260 (+ 111us) server_negotiation.cc:407] Connection header received
0924 15:27:23.534313 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:23.534314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:23.534323 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:23.534394 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:23.534982 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:23.535777 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:23.536786 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:23.537015 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:23.537066 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:23.538501 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:23.538502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:23.538502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:23.538508 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:23.539928 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:23.540328 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:23.540329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:23.540329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:23.540405 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:23.540609 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:23.540610 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:23.540610 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:23.540707 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:23.540793 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:23.540920 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 15:27:23.540981 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:27:23.543007 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26356
2026-09-24T15:27:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:23 dist-test-slave-l766 krb5kdc[26455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:23 dist-test-slave-l766 krb5kdc[26455](info): set up 2 sockets
Sep 24 15:27:23 dist-test-slave-l766 krb5kdc[26455](info): commencing operation
krb5kdc: starting...
W20260924 15:27:24.573150 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:24 dist-test-slave-l766 krb5kdc[26455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263644, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:24.689378 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39903
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:45109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:39903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:24.770520 26468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:24.770769 26468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:24.770808 26468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:24.771982 26468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:24.772059 26468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:24.772083 26468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:24.772104 26468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:24.772132 26468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:24.773645 26468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:45109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:39903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:39903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:24.773960 26468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:24.774199 26468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:24.779212 26474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:24.779229 26476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:24.779374 26468 server_base.cc:1034] running on GCE node
W20260924 15:27:24.779155 26473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:24.779649 26468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:24.779968 26468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:24.781122 26468 hybrid_clock.cc:648] HybridClock initialized: now 1790263644781107 us; error 58 us; skew 500 ppm
Sep 24 15:27:24 dist-test-slave-l766 krb5kdc[26455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263644, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:24.783704 26468 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:24.784178 26468 webserver.cc:460] Webserver started at http://127.4.78.126:43507/ using document root <none> and password file <none>
I20260924 15:27:24.784396 26468 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:24.784462 26468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:24.784610 26468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:24.785640 26468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "7587c990629647ab8f1a393d001e8954"
format_stamp: "Formatted at 2026-09-24 15:27:24 on dist-test-slave-l766"
server_key: "c0096a4bffb1702647f64c80050c732e"
server_key_iv: "ab7105c21f827ca691af6e483a94b776"
server_key_version: "encryptionkey@0"
I20260924 15:27:24.786026 26468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "7587c990629647ab8f1a393d001e8954"
format_stamp: "Formatted at 2026-09-24 15:27:24 on dist-test-slave-l766"
server_key: "c0096a4bffb1702647f64c80050c732e"
server_key_iv: "ab7105c21f827ca691af6e483a94b776"
server_key_version: "encryptionkey@0"
I20260924 15:27:24.787411 26468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:24.788115 26483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:24.788278 26468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:27:24.788384 26468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "7587c990629647ab8f1a393d001e8954"
format_stamp: "Formatted at 2026-09-24 15:27:24 on dist-test-slave-l766"
server_key: "c0096a4bffb1702647f64c80050c732e"
server_key_iv: "ab7105c21f827ca691af6e483a94b776"
server_key_version: "encryptionkey@0"
I20260924 15:27:24.788457 26468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:24.813751 26468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:24.816013 26468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:24.816185 26468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:24.820405 26468 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:39903
I20260924 15:27:24.820453 26535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:39903 every 8 connection(s)
I20260924 15:27:24.820792 26468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:24.821681 26536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:24.824101 26536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954: Bootstrap starting.
I20260924 15:27:24.824498 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26468
I20260924 15:27:24.824627 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:24.824867 4409 external_mini_cluster.cc:1428] Setting key ea234061d59b5a0c6ddc66aa2f265904
I20260924 15:27:24.824869 26536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:24.825169 26536 log.cc:824] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:24.825973 26536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954: No bootstrap required, opened a new log
I20260924 15:27:24.827780 26536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7587c990629647ab8f1a393d001e8954" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39903 } }
I20260924 15:27:24.827936 26536 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:24.827983 26536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7587c990629647ab8f1a393d001e8954, State: Initialized, Role: FOLLOWER
I20260924 15:27:24.828114 26536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7587c990629647ab8f1a393d001e8954" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39903 } }
I20260924 15:27:24.828202 26536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:24.828243 26536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:24.828291 26536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:24.828804 26536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7587c990629647ab8f1a393d001e8954" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39903 } }
I20260924 15:27:24.828946 26536 leader_election.cc:302] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7587c990629647ab8f1a393d001e8954; no voters:
I20260924 15:27:24.829137 26536 leader_election.cc:288] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:24.829160 26541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:24.829295 26541 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [term 1 LEADER]: Becoming Leader. State: Replica: 7587c990629647ab8f1a393d001e8954, State: Running, Role: LEADER
I20260924 15:27:24.829442 26541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7587c990629647ab8f1a393d001e8954" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39903 } }
I20260924 15:27:24.829752 26536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:24.829865 26542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7587c990629647ab8f1a393d001e8954. Latest consensus state: current_term: 1 leader_uuid: "7587c990629647ab8f1a393d001e8954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7587c990629647ab8f1a393d001e8954" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39903 } } }
I20260924 15:27:24.829885 26543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7587c990629647ab8f1a393d001e8954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7587c990629647ab8f1a393d001e8954" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 39903 } } }
I20260924 15:27:24.830000 26542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:24.830011 26543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:24.830247 26545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:24.830684 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:24.825494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58391 (local address 127.4.78.126:39903)
0924 15:27:24.825640 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:24.825649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:24.825681 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:27:24.826197 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:24.826204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:24.826252 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:24.826395 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:24.826625 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:24.827502 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:24.828498 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:24.828943 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:24.829377 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:24.829760 (+ 383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:24.829780 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:24.829781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:24.830017 (+ 236us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:24.830059 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:24.830075 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:24.830392 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:24.830426 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 15:27:24.830491 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":67,"threads_started":1}
I20260924 15:27:24.830943 26545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:24.831717 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:24.832803 26545 catalog_manager.cc:1347] Generated new cluster ID: e451bc4f9efc4928a17592473ab9fa1f
I20260924 15:27:24.832851 26545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:24.840687 26540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:24.833215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57449 (local address 127.4.78.126:39903)
0924 15:27:24.833248 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:24.833250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:24.838080 (+ 4830us) server_negotiation.cc:407] Connection header received
0924 15:27:24.838153 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:24.838154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:24.838164 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:24.838219 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:24.838352 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:24.838812 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:24.839519 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:24.839716 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:24.839787 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:24.840438 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:24.840439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:24.840440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:24.840444 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:24.840457 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:24.840461 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:24.840508 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:24.840570 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:27:24.840589 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:27:24.851936 26545 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:24.852458 26545 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:24.858192 26545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7587c990629647ab8f1a393d001e8954: Generated new TSK 0
I20260924 15:27:24.858376 26545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:24.864004 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26468
2026-09-24T15:27:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1320 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:24 dist-test-slave-l766 krb5kdc[26567](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:24 dist-test-slave-l766 krb5kdc[26567](info): set up 2 sockets
Sep 24 15:27:24 dist-test-slave-l766 krb5kdc[26567](info): commencing operation
krb5kdc: starting...
W20260924 15:27:25.904899 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:25 dist-test-slave-l766 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:26.025484 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:33433
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:33447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:33433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:26.106112 26580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:26.106313 26580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:26.106335 26580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:26.107422 26580 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:26.107453 26580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:26.107467 26580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:26.107481 26580 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:26.107491 26580 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:26.108856 26580 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:33447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:33433
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:33433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:26.109045 26580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:26.109225 26580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:26.113973 26586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:26.113969 26588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:26.114008 26585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:26.114132 26580 server_base.cc:1034] running on GCE node
I20260924 15:27:26.114436 26580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:26.114671 26580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:26.115861 26580 hybrid_clock.cc:648] HybridClock initialized: now 1790263646115837 us; error 36 us; skew 500 ppm
Sep 24 15:27:26 dist-test-slave-l766 krb5kdc[26567](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263646, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:26.118453 26580 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:26.118930 26580 webserver.cc:460] Webserver started at http://127.4.78.126:40085/ using document root <none> and password file <none>
I20260924 15:27:26.119156 26580 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:26.119200 26580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:26.119351 26580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:26.120393 26580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "e242e63087024837a9ad97abfda9315e"
format_stamp: "Formatted at 2026-09-24 15:27:26 on dist-test-slave-l766"
server_key: "bfb0a546576c84e7cef378f8fb78f31e"
server_key_iv: "28d72b0fa12f01baee6663b213c6d1e8"
server_key_version: "encryptionkey@0"
I20260924 15:27:26.120755 26580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "e242e63087024837a9ad97abfda9315e"
format_stamp: "Formatted at 2026-09-24 15:27:26 on dist-test-slave-l766"
server_key: "bfb0a546576c84e7cef378f8fb78f31e"
server_key_iv: "28d72b0fa12f01baee6663b213c6d1e8"
server_key_version: "encryptionkey@0"
I20260924 15:27:26.122100 26580 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:26.122840 26595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:26.123032 26580 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:26.123123 26580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "e242e63087024837a9ad97abfda9315e"
format_stamp: "Formatted at 2026-09-24 15:27:26 on dist-test-slave-l766"
server_key: "bfb0a546576c84e7cef378f8fb78f31e"
server_key_iv: "28d72b0fa12f01baee6663b213c6d1e8"
server_key_version: "encryptionkey@0"
I20260924 15:27:26.123198 26580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:26.126483 26580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:26.128266 26580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:26.128418 26580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:26.132375 26580 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:33433
I20260924 15:27:26.132431 26647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:33433 every 8 connection(s)
I20260924 15:27:26.132754 26580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:26.133668 26648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:26.135901 26648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e: Bootstrap starting.
I20260924 15:27:26.136459 26648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:26.136735 26648 log.cc:824] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:26.137401 26648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e: No bootstrap required, opened a new log
I20260924 15:27:26.139038 26648 raft_consensus.cc:348] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e242e63087024837a9ad97abfda9315e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33433 } }
I20260924 15:27:26.139168 26648 raft_consensus.cc:374] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:27:26.139189 26648 raft_consensus.cc:729] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e242e63087024837a9ad97abfda9315e, State: Initialized, Role: FOLLOWER
I20260924 15:27:26.139369 26648 consensus_queue.cc:260] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e242e63087024837a9ad97abfda9315e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33433 } }
I20260924 15:27:26.139434 26648 raft_consensus.cc:388] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:27:26.139474 26648 raft_consensus.cc:482] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:27:26.139523 26648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:26.139967 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26580
I20260924 15:27:26.140053 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:26.140232 4409 external_mini_cluster.cc:1428] Setting key 959a8f6c7d46aecde4d952d2d152d934
I20260924 15:27:26.140512 26648 raft_consensus.cc:504] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e242e63087024837a9ad97abfda9315e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33433 } }
I20260924 15:27:26.140630 26648 leader_election.cc:302] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e242e63087024837a9ad97abfda9315e; no voters:
I20260924 15:27:26.140836 26648 leader_election.cc:288] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:27:26.140861 26652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:26.141057 26652 raft_consensus.cc:686] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [term 1 LEADER]: Becoming Leader. State: Replica: e242e63087024837a9ad97abfda9315e, State: Running, Role: LEADER
I20260924 15:27:26.141158 26648 sys_catalog.cc:564] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:26.141187 26652 consensus_queue.cc:237] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e242e63087024837a9ad97abfda9315e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33433 } }
I20260924 15:27:26.141624 26652 sys_catalog.cc:455] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e242e63087024837a9ad97abfda9315e. Latest consensus state: current_term: 1 leader_uuid: "e242e63087024837a9ad97abfda9315e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e242e63087024837a9ad97abfda9315e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33433 } } }
I20260924 15:27:26.141772 26652 sys_catalog.cc:458] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:26.141918 26654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e242e63087024837a9ad97abfda9315e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e242e63087024837a9ad97abfda9315e" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 33433 } } }
I20260924 15:27:26.142045 26662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:26.142063 26654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:26.142661 26662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:26.142724 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:26.140751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45751 (local address 127.4.78.126:33433)
0924 15:27:26.140936 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:26.140946 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:26.140977 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:27:26.141116 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:26.141123 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:26.141151 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:26.141604 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:26.142112 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:26.142116 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:26.142116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:26.142120 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:26.142163 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:26.142179 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:26.142466 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:26.142499 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 15:27:26.142541 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":81,"threads_started":1}
I20260924 15:27:26.143819 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:26.144440 26662 catalog_manager.cc:1347] Generated new cluster ID: 73a20aa9eed44cbb94709b751b40788f
I20260924 15:27:26.144491 26662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:26.146818 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:26.145461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40337 (local address 127.4.78.126:33433)
0924 15:27:26.145878 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:26.145880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:26.145887 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:27:26.145907 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:26.145907 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:26.145912 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:26.145964 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:26.146589 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:26.146590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:26.146590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:27:26.146594 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:26.146607 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:27:26.146611 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:26.146664 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:26.146733 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 15:27:26.146745 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402}
I20260924 15:27:26.152379 26662 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:26.152527 26662 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:26.161795 26662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e242e63087024837a9ad97abfda9315e: Generated new TSK 0
I20260924 15:27:26.161993 26662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:26.169922 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26580
2026-09-24T15:27:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1306 ms)
[----------] 18 tests from AuthTokenIssuingTest (23731 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:26 dist-test-slave-l766 krb5kdc[26679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:26 dist-test-slave-l766 krb5kdc[26679](info): set up 2 sockets
Sep 24 15:27:26 dist-test-slave-l766 krb5kdc[26679](info): commencing operation
krb5kdc: starting...
W20260924 15:27:27.200906 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:27.321184 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42659
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:42659,127.4.78.125:37005,127.4.78.124:42281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:27.406090 26692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:27:27.406328 26692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:27.406347 26692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:27.406368 26692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:27.407604 26692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:27.407649 26692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:27.407663 26692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:27.407676 26692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:27.407732 26692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:27.409267 26692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:42659,127.4.78.125:37005,127.4.78.124:42281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:42659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:27.409550 26692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:27.409791 26692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:27.415093 26697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:27.415087 26700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:27.415087 26698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:27.415324 26692 server_base.cc:1034] running on GCE node
I20260924 15:27:27.415598 26692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:27.415915 26692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:27.417089 26692 hybrid_clock.cc:648] HybridClock initialized: now 1790263647417051 us; error 51 us; skew 500 ppm
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:27.424706 26692 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:27.425220 26692 webserver.cc:460] Webserver started at http://127.4.78.126:42211/ using document root <none> and password file <none>
I20260924 15:27:27.425490 26692 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:27.425566 26692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:27.425741 26692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:27.426906 26692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "3d0f01e868564982a324c945aec0ac52"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "8ddd541074dc74152d730f586909869f"
server_key_iv: "4cfee2e75ad522af8ff9a5023d36aa56"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.427305 26692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "3d0f01e868564982a324c945aec0ac52"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "8ddd541074dc74152d730f586909869f"
server_key_iv: "4cfee2e75ad522af8ff9a5023d36aa56"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.428694 26692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:27.429505 26707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:27.429709 26692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:27.429862 26692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "3d0f01e868564982a324c945aec0ac52"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "8ddd541074dc74152d730f586909869f"
server_key_iv: "4cfee2e75ad522af8ff9a5023d36aa56"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.429948 26692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:27.440415 26692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:27.442562 26692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:27.442739 26692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:27.446779 26692 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:42659
I20260924 15:27:27.446815 26759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:42659 every 8 connection(s)
I20260924 15:27:27.447163 26692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:27.448113 26760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:27.449599 26760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:27.455940 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26692
I20260924 15:27:27.456070 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:27.456282 4409 external_mini_cluster.cc:1428] Setting key a7f77e3a5ef65e3f075925724323acb5
I20260924 15:27:27.459113 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.450225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42659 (local address 127.0.0.1:45214)
0924 15:27:27.450372 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.450420 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.450855 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.451105 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.451107 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.451347 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.451513 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.451516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.452483 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.452483 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.453152 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.453153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.453645 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.453646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.453699 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.454280 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.454289 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.455641 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.457574 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.457583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.457586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.457886 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.458142 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.458143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.458143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.458223 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.458676 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.458681 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.458736 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.458815 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.458942 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260924 15:27:27.459218 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.450375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45214 (local address 127.4.78.126:42659)
0924 15:27:27.450501 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.450510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.450538 (+ 28us) server_negotiation.cc:407] Connection header received
0924 15:27:27.450986 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.450988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.451005 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.451081 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.451592 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.452403 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.453233 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.453551 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.453636 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.455745 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.455752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.455753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.455760 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.457447 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.457954 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.457958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.457958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.458071 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.458288 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.458289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.458289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.458603 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.458695 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.458880 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.459031 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260924 15:27:27.459982 26760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:27.460325 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.460204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59220)
0924 15:27:27.460253 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.460296 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:27:27.460461 26709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
I20260924 15:27:27.460878 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.460778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59222)
0924 15:27:27.460833 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.460857 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:27:27.460990 26760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:37005: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.78.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab.
Entry for principal kudu/127.4.78.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab.
I20260924 15:27:27.480933 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.125:37005
--webserver_interface=127.4.78.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab
--principal=kudu/127.4.78.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:42659,127.4.78.125:37005,127.4.78.124:42281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:27:27.510218 26760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } attempt: 1
I20260924 15:27:27.510726 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.510587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59238)
0924 15:27:27.510654 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.510696 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 15:27:27.511098 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.510998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59242)
0924 15:27:27.511045 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.511070 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:27:27.511185 26760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:37005: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
W20260924 15:27:27.563055 26767 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:27:27.563277 26767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:27.563295 26767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:27.563316 26767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:27.564548 26767 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:27.564595 26767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:27.564610 26767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:27.564625 26767 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:27.564638 26767 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:27.566218 26767 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:42659,127.4.78.125:37005,127.4.78.124:42281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab
--principal=kudu/127.4.78.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.125:37005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.78.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:27.566497 26767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:27.566753 26767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:27.571836 26772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:27.571898 26773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:27.571908 26775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:27.572058 26767 server_base.cc:1034] running on GCE node
I20260924 15:27:27.572326 26767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:27.572666 26767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:27.573870 26767 hybrid_clock.cc:648] HybridClock initialized: now 1790263647573825 us; error 48 us; skew 500 ppm
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:27.576436 26767 init.cc:377] Logged in from keytab as kudu/127.4.78.125@KRBTEST.COM (short username kudu)
I20260924 15:27:27.576879 26767 webserver.cc:460] Webserver started at http://127.4.78.125:34269/ using document root <none> and password file <none>
I20260924 15:27:27.577080 26767 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:27.577119 26767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:27.577208 26767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:27.578327 26767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/data/instance:
uuid: "b9782c6618f74abebbb16c2f653af1ba"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "015a33673026ae5c0b2cc8fcc60ef712"
server_key_iv: "b66f9ef782612ceeda3aa46fcedeee31"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.578634 26767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/wal/instance:
uuid: "b9782c6618f74abebbb16c2f653af1ba"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "015a33673026ae5c0b2cc8fcc60ef712"
server_key_iv: "b66f9ef782612ceeda3aa46fcedeee31"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.579325 26760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } attempt: 2
I20260924 15:27:27.579706 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.579577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59254)
0924 15:27:27.579640 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.579677 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:27:27.580071 26767 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:27.580150 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.580029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59268)
0924 15:27:27.580086 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.580122 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 15:27:27.580269 26760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:37005: Network error: Client connection negotiation failed: client connection to 127.4.78.125:37005: connect: Connection refused (error 111)
I20260924 15:27:27.580931 26782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:27.581103 26767 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:27.581205 26767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/wal
uuid: "b9782c6618f74abebbb16c2f653af1ba"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "015a33673026ae5c0b2cc8fcc60ef712"
server_key_iv: "b66f9ef782612ceeda3aa46fcedeee31"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.581290 26767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:27.595265 26767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:27.597714 26767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:27.597927 26767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:27.601802 26767 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.125:37005
I20260924 15:27:27.601878 26834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.125:37005 every 8 connection(s)
I20260924 15:27:27.602281 26767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
I20260924 15:27:27.603253 26835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:27.604812 26835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:27.605906 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26767
I20260924 15:27:27.606034 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-1/wal/instance
I20260924 15:27:27.606207 4409 external_mini_cluster.cc:1428] Setting key 2b70194d1a0c84762106e2d6ec24dd38
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.78.124@KRBTEST.COM; defaulting to no policy
I20260924 15:27:27.615190 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.605531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45226 (local address 127.4.78.126:42659)
0924 15:27:27.605718 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.605720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.605860 (+ 140us) server_negotiation.cc:407] Connection header received
0924 15:27:27.606504 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.606505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.606522 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.606592 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.607211 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.607905 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.609219 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.609579 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.609657 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.612125 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.612127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.612127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.612137 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.613735 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.614187 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.614189 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.614189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.614255 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.614523 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.614524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.614525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.614680 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.614749 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.615052 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.615123 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260924 15:27:27.615346 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.605491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42659 (local address 127.0.0.1:45226)
0924 15:27:27.605679 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.605723 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.606397 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.606663 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.606666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.606913 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.607117 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.607121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.608007 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.608008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.609116 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.609119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.609685 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.609686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.609786 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.610389 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.610402 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.612029 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.613880 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.613887 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.613889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.614111 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.614342 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.614343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.614343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.614444 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.614764 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.614771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.614907 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.614998 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.615127 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":138,"threads_started":1}
I20260924 15:27:27.616034 26835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.124@KRBTEST.COM" created.
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:27:27.624147 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.616331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59282)
0924 15:27:27.616503 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.616512 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.616547 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.616689 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.616689 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.616863 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.616932 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.616934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.617914 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.617916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.618734 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.618736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.619450 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.619450 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.619504 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.619922 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.619926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.621296 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.623089 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.623090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.623090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.623261 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.623479 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.623480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.623480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.623533 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.623826 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.623827 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.623864 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.623936 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.624035 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260924 15:27:27.624208 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.616396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.4.78.125:37005)
0924 15:27:27.616541 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.616546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.616573 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:27:27.616593 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.616594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.616605 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.616665 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.616996 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.617766 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.618845 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.619362 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.619446 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.621401 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.621409 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.621410 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.621416 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.622986 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.623352 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.623355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.623356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.623412 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.623612 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.623613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.623613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.623755 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.623847 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.623973 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.624086 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
I20260924 15:27:27.624835 26835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:27.625203 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.625080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46866)
0924 15:27:27.625128 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.625177 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:27:27.625363 26783 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
I20260924 15:27:27.625752 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.625657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46868)
0924 15:27:27.625707 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.625731 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 15:27:27.625891 26835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:42281: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab.
Entry for principal kudu/127.4.78.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab.
I20260924 15:27:27.633203 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.124:42281
--webserver_interface=127.4.78.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab
--principal=kudu/127.4.78.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:38903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:42659,127.4.78.125:37005,127.4.78.124:42281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:27:27.671451 26760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } attempt: 3
I20260924 15:27:27.675105 26835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } attempt: 1
I20260924 15:27:27.675500 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.675379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46880)
0924 15:27:27.675432 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.675463 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:27:27.675865 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.675741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46890)
0924 15:27:27.675799 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.675833 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 15:27:27.675961 26835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:42281: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:27:27.679298 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.671915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59294 (local address 127.4.78.125:37005)
0924 15:27:27.671995 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.671998 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.672009 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:27:27.672039 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.672040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.672060 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.672124 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.672464 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.672979 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.673965 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.674251 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.674497 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.676711 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.676712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.676713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.676720 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.678153 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.678516 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.678517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.678518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.678576 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.678794 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.678794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.678795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.678907 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.678983 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.679144 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.679214 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 15:27:27.679312 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.671819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59294)
0924 15:27:27.671886 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.671899 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.671971 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.672131 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.672132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.672306 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.672373 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.672375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.673068 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.673070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.673637 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.673639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.674514 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.674515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.674606 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.675147 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.675153 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.676613 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.678247 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.678249 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.678249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.678430 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.678645 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.678646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.678646 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.678705 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.678975 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.678976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.679024 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.679107 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.679210 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 15:27:27.679622 26760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:27.679965 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.679866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46904)
0924 15:27:27.679914 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.679940 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:27:27.680403 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.680313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46906)
0924 15:27:27.680357 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.680381 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:27:27.680528 26760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:42281: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
I20260924 15:27:27.711707 26760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } attempt: 1
I20260924 15:27:27.712250 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.712082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46918)
0924 15:27:27.712162 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.712211 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 15:27:27.712735 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.712624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46924)
0924 15:27:27.712677 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.712711 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:27:27.712891 26760 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:42281: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
W20260924 15:27:27.717465 26842 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:27:27.717670 26842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:27.717689 26842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:27.717736 26842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:27.718961 26842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:27.719002 26842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:27.719017 26842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:27.719030 26842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:27.719043 26842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:27.720559 26842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:38903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:42659,127.4.78.125:37005,127.4.78.124:42281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab
--principal=kudu/127.4.78.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.124:42281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.78.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:27.720822 26842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:27.721057 26842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:27.726143 26848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:27.726126 26847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:27.726125 26850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:27.726158 26842 server_base.cc:1034] running on GCE node
I20260924 15:27:27.726584 26842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:27.726899 26842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:27.728103 26842 hybrid_clock.cc:648] HybridClock initialized: now 1790263647728087 us; error 27 us; skew 500 ppm
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:27.730690 26842 init.cc:377] Logged in from keytab as kudu/127.4.78.124@KRBTEST.COM (short username kudu)
I20260924 15:27:27.731066 26842 webserver.cc:460] Webserver started at http://127.4.78.124:32927/ using document root <none> and password file <none>
I20260924 15:27:27.731253 26842 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:27.731297 26842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:27.731385 26842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:27.732471 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/data/instance:
uuid: "f67b3bf76f37453193bb3cc0c4d146a7"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "428f4958533c4297d0a28b29dfe8ff48"
server_key_iv: "e5a48331eb2ff283724543039ec8a039"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.732844 26842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/wal/instance:
uuid: "f67b3bf76f37453193bb3cc0c4d146a7"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "428f4958533c4297d0a28b29dfe8ff48"
server_key_iv: "e5a48331eb2ff283724543039ec8a039"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.734320 26842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 15:27:27.734993 26857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:27.735177 26842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:27.735307 26842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/wal
uuid: "f67b3bf76f37453193bb3cc0c4d146a7"
format_stamp: "Formatted at 2026-09-24 15:27:27 on dist-test-slave-l766"
server_key: "428f4958533c4297d0a28b29dfe8ff48"
server_key_iv: "e5a48331eb2ff283724543039ec8a039"
server_key_version: "encryptionkey@0"
I20260924 15:27:27.735383 26842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:27.744084 26835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } attempt: 2
I20260924 15:27:27.744503 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.744361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46926)
0924 15:27:27.744443 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.744475 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 15:27:27.744839 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.744743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46938)
0924 15:27:27.744790 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.744817 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:27:27.744935 26835 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:42281: Network error: Client connection negotiation failed: client connection to 127.4.78.124:42281: connect: Connection refused (error 111)
I20260924 15:27:27.766940 26842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:27.769032 26842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:27.769228 26842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:27.773262 26842 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.124:42281
I20260924 15:27:27.773294 26909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.124:42281 every 8 connection(s)
I20260924 15:27:27.773676 26842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
I20260924 15:27:27.774751 26910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:27.776291 26910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:27.778357 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 26842
I20260924 15:27:27.778483 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263382361574-4409-0/minicluster-data/master-2/wal/instance
I20260924 15:27:27.778667 4409 external_mini_cluster.cc:1428] Setting key 68a56372791668bdfa88a103f5c2d562
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:27:27.786244 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.777128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.4.78.126:42659)
0924 15:27:27.777219 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.777222 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.777424 (+ 202us) server_negotiation.cc:407] Connection header received
0924 15:27:27.778005 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.778006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.778024 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.778075 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.778823 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.779437 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.780489 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.780689 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.780779 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.782992 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.782994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.782994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.783002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.784534 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.785031 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.785032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.785033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.785094 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.785450 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.785451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.785451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.785592 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.785671 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.786054 (+ 383us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.786139 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:27:27.786355 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.777067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:42659 (local address 127.0.0.1:45228)
0924 15:27:27.777295 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.777330 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.777887 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.778157 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.778161 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.778437 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.778703 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.778708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.779562 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.779563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.780394 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.780396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.780807 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.780808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.780930 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.781454 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.781463 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.782881 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.784621 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.784628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.784630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.784958 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.785169 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.785169 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.785170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.785354 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.785692 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.785700 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.785890 (+ 190us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.786000 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.786150 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":117,"threads_started":1}
I20260924 15:27:27.787005 26910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:27.787844 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.779085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.4.78.124:42281)
0924 15:27:27.779278 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.779283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.779307 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:27:27.779356 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.779359 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.779377 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.779470 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.780060 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.780784 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.781681 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.782066 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.782153 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.784321 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.784329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.784329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.784336 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.786372 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.786788 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.786791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.786791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.786847 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.787047 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.787048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.787048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.787274 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.787423 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.787666 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.787728 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260924 15:27:27.788056 26760 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } attempt: 2
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:27:27.794806 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.787283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:37005 (local address 127.0.0.1:59308)
0924 15:27:27.787328 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.787337 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.787369 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.787511 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.787512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.787663 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.787697 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.787699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.788480 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.788481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.789266 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.789267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.789726 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.789726 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.789771 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.790198 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.790202 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.791523 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.793282 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.793284 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.793284 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.793608 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.794119 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.794119 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.794120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.794191 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.794495 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.794496 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.794538 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.794608 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.794695 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:27:27.794857 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.787349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.4.78.125:37005)
0924 15:27:27.787391 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.787393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.787402 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:27:27.787426 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.787427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.787442 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.787487 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.787784 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.788331 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.789371 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.789648 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.789715 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.791610 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.791611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.791611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.791619 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.793203 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.793711 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.793712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.793713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.793774 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.794270 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.794271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.794271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.794419 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.794506 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.794646 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.794732 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:27.795440 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.788289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46952)
0924 15:27:27.788354 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.788378 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.788414 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.788566 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.788567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.788735 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.788773 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.788775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.789578 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.789579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.790417 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.790421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.790785 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.790786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.790842 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.791225 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.791230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.792599 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.794438 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.794439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.794440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.794653 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.794865 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.794865 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.794866 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.794935 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.795187 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.795188 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.795230 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.795273 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.795362 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:27.795908 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.788341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.4.78.124:42281)
0924 15:27:27.788401 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.788403 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.788411 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:27.788463 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.788464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.788477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.788556 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.788852 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.789517 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.790496 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.790715 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.790786 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.792682 (+ 1896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.792683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.792684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.792690 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.794364 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.794738 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.794739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.794739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.794790 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.795014 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.795014 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.795015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.795121 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.795699 (+ 578us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.795746 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.795823 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:27:27.795912 26910 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:27.797672 26760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52: Bootstrap starting.
I20260924 15:27:27.798777 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.789209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.4.78.126:42659)
0924 15:27:27.789250 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.789252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.789259 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:27:27.789316 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.789316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.789325 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.789371 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.790338 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.791044 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.793273 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.793552 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.793621 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.796080 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.796082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.796082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.796089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.797581 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.797943 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.797944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.797944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.797996 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.798220 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.798221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.798221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.798326 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.798481 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.798528 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.798609 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:27:27.798774 26760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:27.799170 26760 log.cc:824] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:27.800122 26760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52: No bootstrap required, opened a new log
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:27:27.802474 26760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } }
I20260924 15:27:27.802631 26760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d0f01e868564982a324c945aec0ac52, State: Initialized, Role: FOLLOWER
I20260924 15:27:27.802742 26760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } }
I20260924 15:27:27.803053 26916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.803205 26916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:27.803112 26760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:27.803953 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.796187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46958)
0924 15:27:27.796333 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.796343 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.796383 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.796505 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.796506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.796642 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.796674 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.796676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.797554 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.797555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.798376 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.798379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.799107 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.799108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.799169 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.799542 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.799547 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.800887 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.802931 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.802931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.802932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.803121 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.803310 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.803310 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.803311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.803357 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.803633 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.803634 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.803672 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.803742 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.803844 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260924 15:27:27.804013 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.796295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.4.78.124:42281)
0924 15:27:27.796344 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.796346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.796372 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:27:27.796440 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.796441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.796450 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.796500 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.796751 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.797458 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.798470 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.798784 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.798868 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.800973 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.800974 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.800974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.800980 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.802844 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.803195 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.803196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.803196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.803250 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.803428 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.803429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.803429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.803561 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.803641 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.803770 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.803898 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":32}
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:27:27.805841 26910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7: Bootstrap starting.
W20260924 15:27:27.806195 26927 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:27.806254 26927 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:27.806879 26910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:27.807168 26910 log.cc:824] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:27.807943 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.799699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59322 (local address 127.4.78.125:37005)
0924 15:27:27.799749 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.799751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.799761 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:27.799785 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.799786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.799795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.799854 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.800425 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.800956 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.801760 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.802079 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.802153 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.804653 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.804654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.804655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.804662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.806482 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.806883 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.806883 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.806884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.806941 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.807145 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.807146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.807146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.807271 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.807402 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.807782 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.807853 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:27:27.809408 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:27.807071 26692 master_runner.cc:186] Error getting master registration for 127.4.78.125:37005: OK, instance_id { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" instance_seqno: 1790263647600764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:27:27.810292 26692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:27:27.810524 26734 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3d0f01e868564982a324c945aec0ac52"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45244
I20260924 15:27:27.810611 26734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 15:27:27.810640 26734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:27.811363 26734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } }
I20260924 15:27:27.811602 26734 leader_election.cc:288] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [CANDIDATE]: Term 1 election: Requested vote from peers b9782c6618f74abebbb16c2f653af1ba (127.4.78.125:37005), f67b3bf76f37453193bb3cc0c4d146a7 (127.4.78.124:42281)
I20260924 15:27:27.818542 26910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7: No bootstrap required, opened a new log
I20260924 15:27:27.821033 26910 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } }
I20260924 15:27:27.821203 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.812904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.4.78.126:42659)
0924 15:27:27.812958 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.812960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.813349 (+ 389us) server_negotiation.cc:407] Connection header received
0924 15:27:27.813431 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.813432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.813443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.813493 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.814250 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.814990 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.815999 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.816287 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.816333 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.818027 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.818028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.818028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.818035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.819785 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.820275 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.820277 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.820277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.820337 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.820606 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.820607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.820607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.820741 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.820833 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.821009 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.821087 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:27:27.821208 26910 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67b3bf76f37453193bb3cc0c4d146a7, State: Initialized, Role: FOLLOWER
I20260924 15:27:27.821301 26910 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } }
I20260924 15:27:27.821674 26884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d0f01e868564982a324c945aec0ac52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f67b3bf76f37453193bb3cc0c4d146a7"
I20260924 15:27:27.821797 26884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:27.822769 26884 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0f01e868564982a324c945aec0ac52 in term 1.
I20260924 15:27:27.823021 26708 leader_election.cc:302] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d0f01e868564982a324c945aec0ac52, f67b3bf76f37453193bb3cc0c4d146a7; no voters:
I20260924 15:27:27.821650 26910 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:27.823173 26916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:27.823378 26916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [term 1 LEADER]: Becoming Leader. State: Replica: 3d0f01e868564982a324c945aec0ac52, State: Running, Role: LEADER
I20260924 15:27:27.823477 26916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } }
I20260924 15:27:27.823904 26916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0f01e868564982a324c945aec0ac52. Latest consensus state: current_term: 1 leader_uuid: "3d0f01e868564982a324c945aec0ac52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.823971 26916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:27.824747 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.812708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.4.78.124:42281)
0924 15:27:27.812746 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.812749 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.813206 (+ 457us) server_negotiation.cc:407] Connection header received
0924 15:27:27.813267 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.813268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.813295 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.813354 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.814887 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.815594 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.817654 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.817985 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.818024 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.821228 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.821229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.821230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.821238 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.822912 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.823505 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.823506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.823507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.823572 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.823967 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.823968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.823969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.824092 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.824161 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.824562 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.824650 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:27:27.821637 26937 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.824919 26937 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:27.824941 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.813106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59326 (local address 127.4.78.125:37005)
0924 15:27:27.813154 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.813156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.813511 (+ 355us) server_negotiation.cc:407] Connection header received
0924 15:27:27.813599 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.813600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.813612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.813641 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.814300 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.815016 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.816104 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.816862 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.816907 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.819307 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.819308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.819308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.819315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.820970 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.822918 (+ 1948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.822919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.822919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.822979 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.824354 (+ 1375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.824355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.824355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.824497 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.824574 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.824768 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.824850 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:27:27.826769 26842 master_runner.cc:186] Error getting master registration for 127.4.78.125:37005: OK, instance_id { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" instance_seqno: 1790263647600764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:27:27.829509 26842 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:27:27.830004 26945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:27.830221 26945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 15:27:27.831629 26951 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:27.831682 26951 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:27.831753 26884 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [term 1 FOLLOWER]: Refusing update from remote peer 3d0f01e868564982a324c945aec0ac52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:27.832087 26938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:27.833384 26937 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0f01e868564982a324c945aec0ac52. Latest consensus state: current_term: 1 leader_uuid: "3d0f01e868564982a324c945aec0ac52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.833487 26937 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:27.834355 26916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d0f01e868564982a324c945aec0ac52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.834442 26916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:27.834609 26954 mvcc.cc:205] Tried to move back new op lower bound from 7332919901516537856 to 7332919901485596672. Current Snapshot: MvccSnapshot[applied={T|T < 7332919901516537856}]
I20260924 15:27:27.834685 26937 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d0f01e868564982a324c945aec0ac52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.834764 26937 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:27.834908 26945 catalog_manager.cc:1347] Generated new cluster ID: 9ea6c7c1b393484ba7f18e7a77f6c8b3
I20260924 15:27:27.834955 26945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:27.836061 26835 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } attempt: 3
Sep 24 15:27:27 dist-test-slave-l766 krb5kdc[26679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263647, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:27:27.843709 26837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.836326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:42281 (local address 127.0.0.1:46980)
0924 15:27:27.836436 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:27.836447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:27.836486 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:27.836687 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:27.836687 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:27.836867 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:27.836914 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.836916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.837519 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.837520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.838372 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:27.838374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.838747 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.838747 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:27.838802 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.839241 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:27.839246 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:27.840798 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:27.842602 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.842603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.842603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.842827 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.843029 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:27.843030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:27.843031 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:27.843083 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:27.843384 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:27.843385 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:27.843429 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 15:27:27.843517 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:27:27.843615 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260924 15:27:27.843705 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:27.836557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.4.78.124:42281)
0924 15:27:27.836597 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.836598 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.836606 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:27.836624 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.836625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.836634 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.836683 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.836983 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.837436 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.838440 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:27.838642 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:27.838747 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:27.840894 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:27.840895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:27.840895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:27.840902 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:27.842503 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.842904 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.842905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.842905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.842975 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:27.843171 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:27.843172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:27.843172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:27.843308 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:27.843368 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:27.843543 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 15:27:27.843615 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:27:27.845644 26835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba: Bootstrap starting.
I20260924 15:27:27.846433 26835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:27.846757 26835 log.cc:824] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:27.847726 26835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba: No bootstrap required, opened a new log
I20260924 15:27:27.850371 26945 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:27.850662 26835 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } }
I20260924 15:27:27.850782 26835 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9782c6618f74abebbb16c2f653af1ba, State: Initialized, Role: FOLLOWER
I20260924 15:27:27.850848 26835 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } }
I20260924 15:27:27.851119 26945 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:27.851151 26835 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [sys.catalog]: configured and running, proceeding with master startup.
W20260924 15:27:27.852721 26970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:27.852761 26970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:27.857869 26809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3d0f01e868564982a324c945aec0ac52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9782c6618f74abebbb16c2f653af1ba"
I20260924 15:27:27.857934 26959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.857981 26809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:27.858006 26959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:27.858889 26809 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d0f01e868564982a324c945aec0ac52 in term 1.
I20260924 15:27:27.859285 26808 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [term 1 FOLLOWER]: Refusing update from remote peer 3d0f01e868564982a324c945aec0ac52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:27.859534 26938 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:27.860952 26945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52: Generated new TSK 0
I20260924 15:27:27.861129 26945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:27.861511 26959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d0f01e868564982a324c945aec0ac52. Latest consensus state: current_term: 1 leader_uuid: "3d0f01e868564982a324c945aec0ac52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.861660 26959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:27.861748 26959 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d0f01e868564982a324c945aec0ac52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.861974 26959 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:27.862766 26916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d0f01e868564982a324c945aec0ac52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d0f01e868564982a324c945aec0ac52" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 42659 } } peers { permanent_uuid: "b9782c6618f74abebbb16c2f653af1ba" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 37005 } } peers { permanent_uuid: "f67b3bf76f37453193bb3cc0c4d146a7" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 42281 } } }
I20260924 15:27:27.862854 26916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d0f01e868564982a324c945aec0ac52 [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:27.863934 26972 mvcc.cc:205] Tried to move back new op lower bound from 7332919901577908224 to 7332919901485596672. Current Snapshot: MvccSnapshot[applied={T|T < 7332919901516537856}]
W20260924 15:27:27.899322 26763 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.897552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.4.78.126:42659)
0924 15:27:27.897603 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.897605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.898602 (+ 997us) server_negotiation.cc:407] Connection header received
0924 15:27:27.898711 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.898723 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.898735 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.898766 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.899237 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45260: BlockingRecv error: recv got EOF from 127.0.0.1:45260 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 15:27:27.900319 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.897975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59334 (local address 127.4.78.125:37005)
0924 15:27:27.898017 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.898019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.899691 (+ 1672us) server_negotiation.cc:407] Connection header received
0924 15:27:27.899872 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.899873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.899886 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.899920 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.900256 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59334: BlockingRecv error: recv got EOF from 127.0.0.1:59334 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 15:27:27.901149 26914 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.898314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.4.78.124:42281)
0924 15:27:27.898356 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.898358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.900617 (+ 2259us) server_negotiation.cc:407] Connection header received
0924 15:27:27.900723 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.900724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.900736 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.900765 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.901072 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46984: BlockingRecv error: recv got EOF from 127.0.0.1:46984 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 15:27:27.903724 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.902874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59350 (local address 127.4.78.125:37005)
0924 15:27:27.902908 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.902910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.903165 (+ 255us) server_negotiation.cc:407] Connection header received
0924 15:27:27.903280 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.903281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.903300 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.903329 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.903665 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59350: BlockingRecv error: recv got EOF from 127.0.0.1:59350 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:27:27.906924 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.906201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.4.78.125:37005)
0924 15:27:27.906234 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.906236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.906443 (+ 207us) server_negotiation.cc:407] Connection header received
0924 15:27:27.906535 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.906536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.906545 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.906575 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.906880 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59352: BlockingRecv error: recv got EOF from 127.0.0.1:59352 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:27:27.911005 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.910207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.4.78.125:37005)
0924 15:27:27.910239 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.910241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.910467 (+ 226us) server_negotiation.cc:407] Connection header received
0924 15:27:27.910563 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.910564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.910573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.910602 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.910960 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59358: BlockingRecv error: recv got EOF from 127.0.0.1:59358 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:27:27.916961 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.916212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.4.78.125:37005)
0924 15:27:27.916246 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.916248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.916464 (+ 216us) server_negotiation.cc:407] Connection header received
0924 15:27:27.916559 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.916560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.916570 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.916599 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.916915 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59366: BlockingRecv error: recv got EOF from 127.0.0.1:59366 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 15:27:27.927148 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.926382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.4.78.125:37005)
0924 15:27:27.926420 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.926422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.926641 (+ 219us) server_negotiation.cc:407] Connection header received
0924 15:27:27.926748 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.926749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.926759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.926787 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.927096 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59376: BlockingRecv error: recv got EOF from 127.0.0.1:59376 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:27:27.945755 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.944775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59378 (local address 127.4.78.125:37005)
0924 15:27:27.944821 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.944823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.945109 (+ 286us) server_negotiation.cc:407] Connection header received
0924 15:27:27.945221 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.945223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.945236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.945284 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.945685 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59378: BlockingRecv error: recv got EOF from 127.0.0.1:59378 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 15:27:27.980435 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:27.979414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.4.78.125:37005)
0924 15:27:27.979463 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:27.979465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:27.979809 (+ 344us) server_negotiation.cc:407] Connection header received
0924 15:27:27.979923 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:27.979925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:27.979936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:27.979968 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:27.980373 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59390: BlockingRecv error: recv got EOF from 127.0.0.1:59390 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 15:27:28.046697 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:28.046014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.4.78.125:37005)
0924 15:27:28.046109 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:28.046111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:28.046138 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:27:28.046206 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:28.046206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:28.046217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:28.046268 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:28.046641 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59394: BlockingRecv error: recv got EOF from 127.0.0.1:59394 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 15:27:28.176590 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:28.175865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59404 (local address 127.4.78.125:37005)
0924 15:27:28.176047 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:28.176049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:28.176061 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:27:28.176090 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:28.176091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:28.176101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:28.176164 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:28.176520 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59404: BlockingRecv error: recv got EOF from 127.0.0.1:59404 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
W20260924 15:27:28.434731 26839 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:28.434102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59420 (local address 127.4.78.125:37005)
0924 15:27:28.434172 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:28.434174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:28.434185 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:27:28.434277 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:28.434278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:28.434288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:28.434351 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:28.434680 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59420: BlockingRecv error: recv got EOF from 127.0.0.1:59420 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:27:28.832224 26951 catalog_manager.cc:1259] Loaded cluster ID: 9ea6c7c1b393484ba7f18e7a77f6c8b3
I20260924 15:27:28.832319 26951 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7: loading cluster ID for follower catalog manager: success
I20260924 15:27:28.833129 26951 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7: acquiring CA information for follower catalog manager: success
I20260924 15:27:28.833443 26951 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f67b3bf76f37453193bb3cc0c4d146a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:27:28.853153 26970 catalog_manager.cc:1259] Loaded cluster ID: 9ea6c7c1b393484ba7f18e7a77f6c8b3
I20260924 15:27:28.853245 26970 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba: loading cluster ID for follower catalog manager: success
I20260924 15:27:28.854092 26970 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba: acquiring CA information for follower catalog manager: success
I20260924 15:27:28.854414 26970 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9782c6618f74abebbb16c2f653af1ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:27:28.950838 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:28.948041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.4.78.125:37005)
0924 15:27:28.948169 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:28.948171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:28.948186 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:27:28.948247 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:28.948247 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:28.948298 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:28.948380 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:28.948525 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.949011 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.950080 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.950296 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.950356 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:28.950504 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:28.950669 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:28.950732 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:28.950749 (+ 17us) server_negotiation.cc:299] Negotiation successful
0924 15:27:28.950769 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260924 15:27:28.955192 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:28.952720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.4.78.124:42281)
0924 15:27:28.952868 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:28.952869 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:28.952876 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:27:28.952914 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:28.952915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:28.952959 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:28.953046 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:28.953137 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.953620 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.954417 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.954652 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.954729 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:28.954868 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:28.954972 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:28.955058 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:28.955087 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 15:27:28.955107 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260924 15:27:28.959584 27049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:28.956840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47010 (local address 127.4.78.124:42281)
0924 15:27:28.956877 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:28.956879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:28.957017 (+ 138us) server_negotiation.cc:407] Connection header received
0924 15:27:28.957085 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:28.957086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:28.957096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:28.957145 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:28.957280 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.957721 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.958819 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.959113 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.959181 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:28.959332 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:28.959362 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:28.959411 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:28.959469 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 15:27:28.959489 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:27:28.960019 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:28.956739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45270 (local address 127.4.78.126:42659)
0924 15:27:28.956902 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:28.956904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:28.957009 (+ 105us) server_negotiation.cc:407] Connection header received
0924 15:27:28.957097 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:28.957098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:28.957146 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:28.957226 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:28.957467 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.958057 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.958999 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.959270 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.959328 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:28.959550 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:28.959660 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:28.959727 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:28.959787 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 15:27:28.959934 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20260924 15:27:28.960471 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:28.956931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.4.78.125:37005)
0924 15:27:28.956969 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:28.956970 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:28.957146 (+ 176us) server_negotiation.cc:407] Connection header received
0924 15:27:28.957222 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:28.957223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:28.957234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:28.957283 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:28.958048 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.958533 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.959581 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:28.959940 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:28.959971 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:28.960226 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:28.960256 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:28.960309 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:28.960384 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:27:28.960406 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:28.961654 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26692
I20260924 15:27:28.968096 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26767
I20260924 15:27:28.973313 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 26842
2026-09-24T15:27:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2805 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:27:29 dist-test-slave-l766 krb5kdc[27060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:27:29 dist-test-slave-l766 krb5kdc[27060](info): set up 2 sockets
Sep 24 15:27:29 dist-test-slave-l766 krb5kdc[27060](info): commencing operation
krb5kdc: starting...
W20260924 15:27:30.025134 4409 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/test-user.keytab.
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:27:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:27:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.78.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
Entry for principal kudu/127.4.78.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab.
I20260924 15:27:30.141646 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:44823
--webserver_interface=127.4.78.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:44823,127.4.78.125:42121,127.4.78.124:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:27:30.221032 27073 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:27:30.221292 27073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:30.221329 27073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:30.221374 27073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:30.222517 27073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:30.222584 27073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:30.222607 27073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:30.222636 27073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:30.222668 27073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:30.224128 27073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:44823,127.4.78.125:42121,127.4.78.124:43347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.126.keytab
--principal=kudu/127.4.78.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.126:44823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.78.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:30.224406 27073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:30.224630 27073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:30.229359 27078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:30.229439 27073 server_base.cc:1034] running on GCE node
W20260924 15:27:30.229444 27079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:30.229396 27081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:30.229754 27073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:30.230051 27073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:30.231225 27073 hybrid_clock.cc:648] HybridClock initialized: now 1790263650231214 us; error 38 us; skew 500 ppm
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:30.233632 27073 init.cc:377] Logged in from keytab as kudu/127.4.78.126@KRBTEST.COM (short username kudu)
I20260924 15:27:30.234118 27073 webserver.cc:460] Webserver started at http://127.4.78.126:39601/ using document root <none> and password file <none>
I20260924 15:27:30.234366 27073 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:30.234436 27073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:30.234575 27073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:30.235594 27073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/data/instance:
uuid: "7357758fafb14783b9eff1dedbb0f4ee"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "bd3b9a99a58ac883081357c5e83ece8b"
server_key_iv: "c3dbe1926f4a7d337a017a49429008ae"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.235948 27073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/wal/instance:
uuid: "7357758fafb14783b9eff1dedbb0f4ee"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "bd3b9a99a58ac883081357c5e83ece8b"
server_key_iv: "c3dbe1926f4a7d337a017a49429008ae"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.237244 27073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:27:30.238089 27088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:30.238247 27073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:27:30.238376 27073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/wal
uuid: "7357758fafb14783b9eff1dedbb0f4ee"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "bd3b9a99a58ac883081357c5e83ece8b"
server_key_iv: "c3dbe1926f4a7d337a017a49429008ae"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.238451 27073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:30.253014 27073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:30.254963 27073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:30.255120 27073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:30.259176 27073 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.126:44823
I20260924 15:27:30.259212 27140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.126:44823 every 8 connection(s)
I20260924 15:27:30.259564 27073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/data/info.pb
I20260924 15:27:30.260514 27141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:30.262115 27141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:30.265983 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 27073
I20260924 15:27:30.266063 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-0/wal/instance
I20260924 15:27:30.266244 4409 external_mini_cluster.cc:1428] Setting key 9711b0b38fa0e2a922397defc214e4a1
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:30.271680 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.262744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:44823 (local address 127.0.0.1:55208)
0924 15:27:30.262911 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.262953 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.263402 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.263640 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.263643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.263885 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.264033 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.264037 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.264993 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.264994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.265655 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.265657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.266315 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.266316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.266388 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.266861 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.266869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.268314 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.270188 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.270194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.270196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.270394 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.270645 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.270645 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.270646 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.270734 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.271079 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.271083 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.271143 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.271254 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.271492 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":98,"thread_start_us":71,"threads_started":1}
I20260924 15:27:30.271684 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.262811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55208 (local address 127.4.78.126:44823)
0924 15:27:30.262985 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.262989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.263044 (+ 55us) server_negotiation.cc:407] Connection header received
0924 15:27:30.263522 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.263523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.263540 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.263612 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.264108 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.264911 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.265722 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.266165 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.266304 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.268405 (+ 2101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.268412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.268413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.268420 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.270072 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.270490 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.270493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.270493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.270575 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.270801 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.270802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.270802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.271006 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.271099 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.271286 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.271446 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260924 15:27:30.272666 27141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:30.273005 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.272891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56010)
0924 15:27:30.272944 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.272985 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 15:27:30.273140 27091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.4.78.125@KRBTEST.COM; defaulting to no policy
I20260924 15:27:30.273540 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.273445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56014)
0924 15:27:30.273493 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.273517 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:27:30.273641 27141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:42121: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab.
Entry for principal kudu/127.4.78.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab.
I20260924 15:27:30.292547 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.125:42121
--webserver_interface=127.4.78.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab
--principal=kudu/127.4.78.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:44823,127.4.78.125:42121,127.4.78.124:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:27:30.322876 27141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } attempt: 1
I20260924 15:27:30.323365 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.323221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56016)
0924 15:27:30.323302 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.323336 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 15:27:30.323704 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.323606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56030)
0924 15:27:30.323653 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.323679 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:27:30.323796 27141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:42121: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
W20260924 15:27:30.381093 27148 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:27:30.381311 27148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:30.381328 27148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:30.381348 27148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:30.382573 27148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:30.382615 27148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:30.382633 27148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:30.382686 27148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:30.382714 27148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:30.384233 27148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:44823,127.4.78.125:42121,127.4.78.124:43347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.125.keytab
--principal=kudu/127.4.78.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.125:42121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.78.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:30.384486 27148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:30.384727 27148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:30.389524 27156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:30.389492 27154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:30.389482 27153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:30.389739 27148 server_base.cc:1034] running on GCE node
I20260924 15:27:30.389988 27148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:30.390275 27148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:30.391418 27148 hybrid_clock.cc:648] HybridClock initialized: now 1790263650391395 us; error 39 us; skew 500 ppm
I20260924 15:27:30.391927 27141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } attempt: 2
I20260924 15:27:30.392343 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.392191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56044)
0924 15:27:30.392263 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.392304 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 15:27:30.392688 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.392586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56054)
0924 15:27:30.392637 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.392667 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:27:30.392818 27141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.125:42121: Network error: Client connection negotiation failed: client connection to 127.4.78.125:42121: connect: Connection refused (error 111)
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:30.394112 27148 init.cc:377] Logged in from keytab as kudu/127.4.78.125@KRBTEST.COM (short username kudu)
I20260924 15:27:30.394556 27148 webserver.cc:460] Webserver started at http://127.4.78.125:41741/ using document root <none> and password file <none>
I20260924 15:27:30.394790 27148 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:30.394860 27148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:30.395015 27148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:30.396029 27148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/data/instance:
uuid: "196add3fbb184da5842c43bb9480588d"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "0ab7a19a02217c733014f373e5498490"
server_key_iv: "62f98a4e7b28d8b2bd6a346144738959"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.396392 27148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/wal/instance:
uuid: "196add3fbb184da5842c43bb9480588d"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "0ab7a19a02217c733014f373e5498490"
server_key_iv: "62f98a4e7b28d8b2bd6a346144738959"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.397718 27148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:27:30.398499 27163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:30.398687 27148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:30.398799 27148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/wal
uuid: "196add3fbb184da5842c43bb9480588d"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "0ab7a19a02217c733014f373e5498490"
server_key_iv: "62f98a4e7b28d8b2bd6a346144738959"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.398890 27148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:30.414597 27148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:30.416433 27148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:30.416594 27148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:30.420727 27148 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.125:42121
I20260924 15:27:30.420773 27215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.125:42121 every 8 connection(s)
I20260924 15:27:30.421139 27148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/data/info.pb
I20260924 15:27:30.422164 27216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:30.423660 27216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:30.429160 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 27148
I20260924 15:27:30.429266 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-1/wal/instance
I20260924 15:27:30.429467 4409 external_mini_cluster.cc:1428] Setting key 209d8bb0280b56591a3ed959cf63aeba
I20260924 15:27:30.434945 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.424349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.4.78.126:44823)
0924 15:27:30.424419 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.424421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.424607 (+ 186us) server_negotiation.cc:407] Connection header received
0924 15:27:30.425062 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.425063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.425075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.425135 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.425726 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.426260 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.427197 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.427401 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.427458 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.429775 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.429776 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.429776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.429783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.433449 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.433920 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.433922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.433923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.433991 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.434232 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.434233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.434233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.434398 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.434475 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.434740 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.434828 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:27:30.434968 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.424307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:44823 (local address 127.0.0.1:55220)
0924 15:27:30.424476 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.424517 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.424974 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.425211 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.425215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.425453 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.425639 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.425642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.426345 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.426346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.427119 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.427121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.427515 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.427516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.427617 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.428052 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.428060 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.429659 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.433542 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.433549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.433551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.433773 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.434064 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.434064 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.434065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.434164 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.434474 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.434479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.434596 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.434689 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.434811 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260924 15:27:30.435603 27216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.78.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.78.124@KRBTEST.COM" created.
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:27:30.443830 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.435877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56060)
0924 15:27:30.435925 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.435936 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.435977 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.436228 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.436229 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.436381 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.436430 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.436432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.437312 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.437313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.437898 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.437901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.438387 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.438388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.438450 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.438807 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.438811 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.440561 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.442554 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.442555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.442556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.442793 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.442993 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.442994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.442995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.443065 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.443364 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.443365 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.443415 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.443497 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.443617 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":1}
I20260924 15:27:30.443840 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.435945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56060 (local address 127.4.78.125:42121)
0924 15:27:30.436099 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.436103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.436119 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:27:30.436136 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.436138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.436148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.436272 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.436508 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.437229 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.437988 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.438299 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.438388 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.440631 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.440641 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.440642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.440650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.442468 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.442873 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.442876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.442876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.442927 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.443124 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.443125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.443125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.443290 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.443354 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.443520 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.443703 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":93,"server-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260924 15:27:30.444533 27216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:30.444875 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.444783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36520)
0924 15:27:30.444816 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.444856 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 15:27:30.445042 27164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
I20260924 15:27:30.445484 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.445381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36528)
0924 15:27:30.445433 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.445460 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 15:27:30.445616 27216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:43347: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.78.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab.
Entry for principal kudu/127.4.78.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab.
I20260924 15:27:30.456485 4409 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskwmKTEk/build/release/bin/kudu
/tmp/dist-test-taskwmKTEk/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.124:43347
--webserver_interface=127.4.78.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab
--principal=kudu/127.4.78.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.78.84:37967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.78.126:44823,127.4.78.125:42121,127.4.78.124:43347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:27:30.483985 27141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } attempt: 3
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:27:30.491593 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.484354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56066)
0924 15:27:30.484431 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.484444 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.484501 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.484673 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.484673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.484824 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.484892 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.484893 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.485859 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.485861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.486419 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.486421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.486950 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.486952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.487018 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.487496 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.487502 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.488700 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.490496 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.490497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.490498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.490683 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.490910 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.490911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.490912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.490974 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.491258 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.491259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.491316 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.491392 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.491495 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 15:27:30.491593 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.484454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56066 (local address 127.4.78.125:42121)
0924 15:27:30.484522 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.484524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.484535 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:27:30.484566 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.484567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.484581 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.484659 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.484973 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.485730 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.486506 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.486863 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.486939 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.488813 (+ 1874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.488815 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.488815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.488823 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.490389 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.490779 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.490781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.490781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.490841 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.491040 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.491041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.491041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.491172 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.491264 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.491419 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.491495 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:27:30.492004 27141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:30.492349 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.492253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36534)
0924 15:27:30.492300 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.492330 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:27:30.492650 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.492559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36546)
0924 15:27:30.492607 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.492630 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:27:30.492748 27141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:43347: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
I20260924 15:27:30.494863 27216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } attempt: 1
I20260924 15:27:30.495240 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.495127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36548)
0924 15:27:30.495179 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.495213 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:27:30.495580 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.495484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36556)
0924 15:27:30.495532 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.495558 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:27:30.495692 27216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:43347: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
I20260924 15:27:30.523903 27141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } attempt: 1
I20260924 15:27:30.524431 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.524279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36570)
0924 15:27:30.524360 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.524403 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 15:27:30.524847 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.524720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36586)
0924 15:27:30.524795 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.524825 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 15:27:30.524941 27141 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:43347: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
W20260924 15:27:30.539335 27223 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:27:30.539566 27223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:27:30.539608 27223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:27:30.539644 27223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:27:30.540853 27223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:27:30.540913 27223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:27:30.540938 27223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:27:30.540978 27223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:27:30.541008 27223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:27:30.542586 27223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.78.84:37967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.78.126:44823,127.4.78.125:42121,127.4.78.124:43347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/krb5kdc/kudu_127.4.78.124.keytab
--principal=kudu/127.4.78.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.78.124:43347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.78.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:27:30.542860 27223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:27:30.543077 27223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:27:30.547744 27228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:30.547816 27231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:27:30.547749 27229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:27:30.548010 27223 server_base.cc:1034] running on GCE node
I20260924 15:27:30.548316 27223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:27:30.548615 27223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:27:30.549779 27223 hybrid_clock.cc:648] HybridClock initialized: now 1790263650549765 us; error 40 us; skew 500 ppm
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:27:30.552183 27223 init.cc:377] Logged in from keytab as kudu/127.4.78.124@KRBTEST.COM (short username kudu)
I20260924 15:27:30.552681 27223 webserver.cc:460] Webserver started at http://127.4.78.124:36449/ using document root <none> and password file <none>
I20260924 15:27:30.552899 27223 fs_manager.cc:359] Metadata directory not provided
I20260924 15:27:30.552966 27223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:27:30.553113 27223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:27:30.554212 27223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/data/instance:
uuid: "095cc4c2e7ec485ea16a629aab2df407"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "9f305d5fb6de223a8b6374b3ac2fe770"
server_key_iv: "06a773b5ff220d4e13402ff0dba9f0e0"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.554577 27223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/wal/instance:
uuid: "095cc4c2e7ec485ea16a629aab2df407"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "9f305d5fb6de223a8b6374b3ac2fe770"
server_key_iv: "06a773b5ff220d4e13402ff0dba9f0e0"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.555914 27223 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:27:30.556654 27238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:27:30.556821 27223 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:27:30.556924 27223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/data,/tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/wal
uuid: "095cc4c2e7ec485ea16a629aab2df407"
format_stamp: "Formatted at 2026-09-24 15:27:30 on dist-test-slave-l766"
server_key: "9f305d5fb6de223a8b6374b3ac2fe770"
server_key_iv: "06a773b5ff220d4e13402ff0dba9f0e0"
server_key_version: "encryptionkey@0"
I20260924 15:27:30.556998 27223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:27:30.563823 27216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } attempt: 2
I20260924 15:27:30.564210 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.564090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36600)
0924 15:27:30.564147 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.564182 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 15:27:30.564566 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.564468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36616)
0924 15:27:30.564516 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.564548 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:27:30.564647 27216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.78.124:43347: Network error: Client connection negotiation failed: client connection to 127.4.78.124:43347: connect: Connection refused (error 111)
I20260924 15:27:30.577714 27223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:27:30.579937 27223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:27:30.580117 27223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:27:30.584321 27223 rpc_server.cc:307] RPC server started. Bound to: 127.4.78.124:43347
I20260924 15:27:30.584354 27290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.78.124:43347 every 8 connection(s)
I20260924 15:27:30.584729 27223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/data/info.pb
I20260924 15:27:30.585707 27291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:27:30.587311 27291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:30.591126 4409 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskwmKTEk/build/release/bin/kudu as pid 27223
I20260924 15:27:30.591221 4409 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskwmKTEk/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263382361574-4409-0/minicluster-data/master-2/wal/instance
I20260924 15:27:30.591419 4409 external_mini_cluster.cc:1428] Setting key b51a77759cf40810a1495e998605cd5a
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:30.596597 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.587964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.4.78.126:44823)
0924 15:27:30.588052 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.588054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.588180 (+ 126us) server_negotiation.cc:407] Connection header received
0924 15:27:30.588655 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.588656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.588667 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.588714 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.589289 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.589746 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.590760 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.591002 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.591081 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.593537 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.593539 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.593540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.593548 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.595139 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.595605 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.595606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.595606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.595656 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.595945 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.595946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.595946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.596044 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.596094 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.596422 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.596486 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 15:27:30.596827 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.587913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.126:44823 (local address 127.0.0.1:55234)
0924 15:27:30.588067 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.588103 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.588570 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.588773 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.588776 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.589016 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.589221 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.589225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.589884 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.589885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.590682 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.590684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.591110 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.591111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.591245 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.591994 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.592012 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.593451 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.595210 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.595217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.595219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.595526 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.595723 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.595724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.595725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.595872 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.596111 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.596117 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.596281 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.596403 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.596566 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:27:30.597334 27291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:27:30.600157 27141 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } attempt: 2
I20260924 15:27:30.601521 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.591788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36626 (local address 127.4.78.124:43347)
0924 15:27:30.591945 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.591950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.591995 (+ 45us) server_negotiation.cc:407] Connection header received
0924 15:27:30.592041 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.592043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.592067 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.592152 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.592711 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.593507 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.594447 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.594957 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.595034 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.597699 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.597706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.597706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.597714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.599539 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.599956 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.599959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.599959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.600019 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.600237 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.600237 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.600238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.600691 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.600880 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.601318 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.601399 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.126@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:27:30.605667 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.597576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.125:42121 (local address 127.0.0.1:56080)
0924 15:27:30.597621 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.597633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.597668 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.597866 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.597867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.597991 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.598033 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.598034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.598843 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.598844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.599633 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.599635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.600115 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.600116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.600186 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.600602 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.600607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.602024 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.603921 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.603922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.603922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.604128 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.604333 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.604333 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.604334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.604390 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.605229 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.605230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.605311 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.605438 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.605567 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 15:27:30.605696 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.597663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.4.78.125:42121)
0924 15:27:30.597709 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.597712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.597720 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:30.597743 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.597744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.597754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.597838 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.598112 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.598741 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.599751 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.600012 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.600114 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.602123 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.602124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.602125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.602132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.603832 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.604183 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.604184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.604185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.604237 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.605032 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.605033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.605033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.605174 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.605236 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.605498 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.605581 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:27:30.606137 27291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.126@KRBTEST.COM
I20260924 15:27:30.609242 27143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.600393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36632)
0924 15:27:30.600446 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.600456 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.600494 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.600747 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.600749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.600913 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.600953 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.600955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.602120 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.602121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.602892 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.602895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.603429 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.603430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.603491 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.603898 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.603902 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.605296 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.608193 (+ 2897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.608194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.608194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.608377 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.608598 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.608598 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.608599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.608661 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.608918 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.608918 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.608961 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.609046 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.609136 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:27:30.609262 27296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.600450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.4.78.124:43347)
0924 15:27:30.600614 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.600616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.600623 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:27:30.600641 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.600642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.600652 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.600722 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.601039 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.602022 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.602994 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.603350 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.603411 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.605392 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.605393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.605393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.605399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.608091 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.608465 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.608466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.608466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.608520 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.608730 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.608731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.608731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.608843 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.608913 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.609071 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.609141 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.124@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:27:30.611462 27141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee: Bootstrap starting.
I20260924 15:27:30.611552 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.602296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55240 (local address 127.4.78.126:44823)
0924 15:27:30.602349 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.602351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.602656 (+ 305us) server_negotiation.cc:407] Connection header received
0924 15:27:30.602677 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.602678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.602689 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.602751 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.603327 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.604018 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.604797 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.605087 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.605157 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.608157 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.608158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.608158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.608165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.609986 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.610353 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.610355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.610356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.610413 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.610632 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.610633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.610633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.610812 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.610968 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.611040 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.611112 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:30.612139 27141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:30.612470 27141 log.cc:824] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:30.613312 27141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee: No bootstrap required, opened a new log
I20260924 15:27:30.614164 27293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.606393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36642)
0924 15:27:30.606431 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.606440 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.606476 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.606597 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.606598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.606726 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.606749 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.606750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.607457 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.607458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.607980 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.607982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.608753 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.608754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.608823 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.609614 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.609620 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.610958 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.613057 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.613058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.613058 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.613263 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.613452 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.613453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.613453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.613519 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.613902 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.613902 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.613945 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.614020 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.614112 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 15:27:30.615056 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.606423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36642 (local address 127.4.78.124:43347)
0924 15:27:30.606453 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.606455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.606465 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:30.606531 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.606531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.606541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.606594 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.606838 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.607379 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.608059 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.608642 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.608707 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.611039 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.611040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.611040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.611047 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.612968 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.613345 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.613346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.613346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.613395 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.613654 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.613655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.613655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.613840 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.613907 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.614900 (+ 993us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.614976 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:27:30.615563 27141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } }
I20260924 15:27:30.615707 27141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7357758fafb14783b9eff1dedbb0f4ee, State: Initialized, Role: FOLLOWER
I20260924 15:27:30.615782 27141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } }
I20260924 15:27:30.616060 27298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.616119 27141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:30.616551 27291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407: Bootstrap starting.
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.78.125@KRBTEST.COM
I20260924 15:27:30.617301 27291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:30.617636 27291 log.cc:824] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:30.616207 27298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:27:30.618273 27309 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:30.618396 27309 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:30.618557 27291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407: No bootstrap required, opened a new log
I20260924 15:27:30.620388 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.612592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.4.78.125:42121)
0924 15:27:30.612630 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.612632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.612639 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:27:30.612659 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.612660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.612668 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.612745 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.613321 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.613965 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.614854 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.615036 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.615120 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.617653 (+ 2533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.617655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.617656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.617663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.619291 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.619629 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.619630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.619630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.619690 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.619900 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.619901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.619901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.620008 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.620155 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.620220 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.620293 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:30.620747 27291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } }
I20260924 15:27:30.620900 27291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095cc4c2e7ec485ea16a629aab2df407, State: Initialized, Role: FOLLOWER
I20260924 15:27:30.620990 27291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } }
I20260924 15:27:30.621238 27312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.621297 27291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:30.619061 27073 master_runner.cc:186] Error getting master registration for 127.4.78.125:42121: OK, instance_id { permanent_uuid: "196add3fbb184da5842c43bb9480588d" instance_seqno: 1790263650419605 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:27:30.621910 27073 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:27:30.622473 4409 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:27:30.622710 27312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:30.623471 27115 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7357758fafb14783b9eff1dedbb0f4ee"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55240
I20260924 15:27:30.623613 27115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 15:27:30.623661 27115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:30.624565 27115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } }
I20260924 15:27:30.624822 27115 leader_election.cc:288] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [CANDIDATE]: Term 1 election: Requested vote from peers 196add3fbb184da5842c43bb9480588d (127.4.78.125:42121), 095cc4c2e7ec485ea16a629aab2df407 (127.4.78.124:43347)
I20260924 15:27:30.625051 27265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7357758fafb14783b9eff1dedbb0f4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "095cc4c2e7ec485ea16a629aab2df407"
I20260924 15:27:30.625165 27265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:30.626163 27265 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7357758fafb14783b9eff1dedbb0f4ee in term 1.
W20260924 15:27:30.626523 27323 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:27:30.626713 27323 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:27:30.626878 27089 leader_election.cc:302] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 095cc4c2e7ec485ea16a629aab2df407, 7357758fafb14783b9eff1dedbb0f4ee; no voters:
I20260924 15:27:30.627010 27298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:27:30.627198 27298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [term 1 LEADER]: Becoming Leader. State: Replica: 7357758fafb14783b9eff1dedbb0f4ee, State: Running, Role: LEADER
I20260924 15:27:30.627302 27298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } }
I20260924 15:27:30.628319 27332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7357758fafb14783b9eff1dedbb0f4ee. Latest consensus state: current_term: 1 leader_uuid: "7357758fafb14783b9eff1dedbb0f4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.628423 27332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:30.629285 27223 master_runner.cc:186] Error getting master registration for 127.4.78.125:42121: OK, instance_id { permanent_uuid: "196add3fbb184da5842c43bb9480588d" instance_seqno: 1790263650419605 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:27:30.631810 27223 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:27:30.634752 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.626099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36654 (local address 127.4.78.124:43347)
0924 15:27:30.626138 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.626140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.626890 (+ 750us) server_negotiation.cc:407] Connection header received
0924 15:27:30.626986 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.626988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.626998 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.627046 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.627853 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.628565 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.629739 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.630060 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.630104 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.631691 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.631692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.631693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.631700 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.633368 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.633800 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.633801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.633802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.633894 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.634124 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.634124 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.634125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.634251 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.634337 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.634572 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.634646 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:30.637429 27334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:27:30.637719 27334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:27:30.639668 27265 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [term 1 FOLLOWER]: Refusing update from remote peer 7357758fafb14783b9eff1dedbb0f4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:30.639997 27298 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:30.641984 27312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7357758fafb14783b9eff1dedbb0f4ee. Latest consensus state: current_term: 1 leader_uuid: "7357758fafb14783b9eff1dedbb0f4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.642084 27312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:30.644161 27144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.626630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55246 (local address 127.4.78.126:44823)
0924 15:27:30.626666 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.626668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.634225 (+ 7557us) server_negotiation.cc:407] Connection header received
0924 15:27:30.634367 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.634368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.634378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.634407 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.635604 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.636344 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.638513 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.638828 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.638867 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.640673 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.640675 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.640675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.640682 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.642328 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.643047 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.643048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.643048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.643102 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.643425 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.643426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.643426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.643549 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.643629 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.643997 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.644073 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:27:30.645327 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.626626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56104 (local address 127.4.78.125:42121)
0924 15:27:30.626736 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.626738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.633990 (+ 7252us) server_negotiation.cc:407] Connection header received
0924 15:27:30.634101 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.634102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.634115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.634167 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.635325 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.636035 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.636992 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.637297 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.637371 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.642420 (+ 5049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.642421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.642421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.642429 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.644068 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.644471 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.644472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.644473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.644531 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.644733 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.644734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.644734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.644856 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.644938 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.645146 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.645226 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 15:27:30.646409 27298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7357758fafb14783b9eff1dedbb0f4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.646490 27298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:30.646919 27312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7357758fafb14783b9eff1dedbb0f4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.647008 27312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:30.647359 27334 catalog_manager.cc:1347] Generated new cluster ID: f14d990529174ed7b5d0aaa346e5db9f
I20260924 15:27:30.647413 27334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:27:30.648691 27336 mvcc.cc:205] Tried to move back new op lower bound from 7332919913016750080 to 7332919912970108928. Current Snapshot: MvccSnapshot[applied={T|T < 7332919913016750080}]
I20260924 15:27:30.655748 27216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } attempt: 3
Sep 24 15:27:30 dist-test-slave-l766 krb5kdc[27060](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263650, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.78.125@KRBTEST.COM for kudu/127.4.78.124@KRBTEST.COM
I20260924 15:27:30.664893 27218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.656069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.78.124:43347 (local address 127.0.0.1:36666)
0924 15:27:30.656150 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 15:27:30.656161 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:27:30.656196 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:27:30.656436 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:27:30.656437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:27:30.656607 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:27:30.656655 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.656657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.657570 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.657571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.658420 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:27:30.658423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.659006 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.659007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:27:30.659078 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.659548 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:27:30.659553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:27:30.661766 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:27:30.663719 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.663720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.663720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.663947 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.664189 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:27:30.664190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:27:30.664190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:27:30.664257 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:27:30.664576 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:27:30.664577 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:27:30.664634 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 15:27:30.664694 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:27:30.664796 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 15:27:30.665416 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.656275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36666 (local address 127.4.78.124:43347)
0924 15:27:30.656313 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.656315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.656321 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:27:30.656341 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.656342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.656351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.656395 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.656731 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.657479 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.658530 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.658866 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.658923 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.661947 (+ 3024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:27:30.661949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:27:30.661949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:27:30.661956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:27:30.663610 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.664039 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.664040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.664041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.664109 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:27:30.664335 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:27:30.664336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:27:30.664336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:27:30.664476 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:27:30.664533 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.665255 (+ 722us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.665328 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:27:30.667279 27216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d: Bootstrap starting.
I20260924 15:27:30.668015 27216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d: Neither blocks nor log segments found. Creating new log.
I20260924 15:27:30.668306 27216 log.cc:824] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:27:30.668691 27334 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:27:30.669427 27334 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:27:30.673503 27334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee: Generated new TSK 0
I20260924 15:27:30.673725 27334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:27:30.673741 27216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d: No bootstrap required, opened a new log
I20260924 15:27:30.676278 27216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } }
I20260924 15:27:30.676421 27216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 196add3fbb184da5842c43bb9480588d, State: Initialized, Role: FOLLOWER
I20260924 15:27:30.676503 27216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } }
I20260924 15:27:30.676877 27216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:27:30.676960 27189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:27:30.677899 27189 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [term 1 FOLLOWER]: Refusing update from remote peer 7357758fafb14783b9eff1dedbb0f4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:27:30.678226 27332 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:27:30.679670 27190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7357758fafb14783b9eff1dedbb0f4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "196add3fbb184da5842c43bb9480588d"
I20260924 15:27:30.676851 27341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.680976 27341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:30.681041 27190 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7357758fafb14783b9eff1dedbb0f4ee in term 1.
I20260924 15:27:30.682782 27341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7357758fafb14783b9eff1dedbb0f4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.682873 27341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:30.686198 27356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7357758fafb14783b9eff1dedbb0f4ee. Latest consensus state: current_term: 1 leader_uuid: "7357758fafb14783b9eff1dedbb0f4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:30.686872 27356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:27:30.686823 27353 catalog_manager.cc:1259] Loaded cluster ID: f14d990529174ed7b5d0aaa346e5db9f
I20260924 15:27:30.686991 27353 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d: loading cluster ID for follower catalog manager: success
I20260924 15:27:30.688225 27353 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d: acquiring CA information for follower catalog manager: success
I20260924 15:27:30.688616 27353 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 196add3fbb184da5842c43bb9480588d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 15:27:30.705861 27144 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.704479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55262 (local address 127.4.78.126:44823)
0924 15:27:30.704526 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.704528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.705156 (+ 628us) server_negotiation.cc:407] Connection header received
0924 15:27:30.705262 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.705263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.705275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.705305 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.705755 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55262: BlockingRecv error: recv got EOF from 127.0.0.1:55262 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 15:27:30.706775 27220 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.704860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56114 (local address 127.4.78.125:42121)
0924 15:27:30.704899 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.704901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.706265 (+ 1364us) server_negotiation.cc:407] Connection header received
0924 15:27:30.706360 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.706361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.706372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.706400 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.706716 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56114: BlockingRecv error: recv got EOF from 127.0.0.1:56114 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:27:30.707803 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.707225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.4.78.124:43347)
0924 15:27:30.707267 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.707269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.707306 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:27:30.707398 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.707399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.707409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.707437 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.707742 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36682: BlockingRecv error: recv got EOF from 127.0.0.1:36682 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:27:30.712879 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.709434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.4.78.125:42121)
0924 15:27:30.709466 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.709468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.709646 (+ 178us) server_negotiation.cc:407] Connection header received
0924 15:27:30.709762 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.709763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.709805 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.709873 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:30.710073 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.710857 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.711973 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.712266 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.712303 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.712503 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:30.712627 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:30.712674 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.712793 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.712811 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:27:30.716161 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.715416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36690 (local address 127.4.78.124:43347)
0924 15:27:30.715454 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.715456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.715607 (+ 151us) server_negotiation.cc:407] Connection header received
0924 15:27:30.715728 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.715729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.715753 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.715786 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.716107 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36690: BlockingRecv error: recv got EOF from 127.0.0.1:36690 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:30.721602 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.718542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56138 (local address 127.4.78.125:42121)
0924 15:27:30.718576 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.718578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.718827 (+ 249us) server_negotiation.cc:407] Connection header received
0924 15:27:30.718919 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.718920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.718929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.718955 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:30.719108 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.719755 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.720786 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.721035 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.721076 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.721324 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:30.721356 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:30.721384 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.721491 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.721511 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:27:30.725103 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.724304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.4.78.124:43347)
0924 15:27:30.724345 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.724347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.724544 (+ 197us) server_negotiation.cc:407] Connection header received
0924 15:27:30.724646 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.724647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.724659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.724703 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.725056 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36694: BlockingRecv error: recv got EOF from 127.0.0.1:36694 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:27:30.731417 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.728274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56152 (local address 127.4.78.125:42121)
0924 15:27:30.728308 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.728309 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.728532 (+ 223us) server_negotiation.cc:407] Connection header received
0924 15:27:30.728621 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.728622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.728631 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.728657 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:30.728868 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.729551 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.730641 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.730923 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.730967 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.731166 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:30.731199 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:30.731228 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.731335 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.731353 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:27:30.734586 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.733747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.4.78.124:43347)
0924 15:27:30.733785 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.733786 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.734063 (+ 277us) server_negotiation.cc:407] Connection header received
0924 15:27:30.734160 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.734161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.734173 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.734203 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.734539 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36698: BlockingRecv error: recv got EOF from 127.0.0.1:36698 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:30.743295 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.739896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.4.78.125:42121)
0924 15:27:30.739936 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.739938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.740208 (+ 270us) server_negotiation.cc:407] Connection header received
0924 15:27:30.740309 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.740311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.740321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.740349 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:30.740542 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.741296 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.742474 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.742766 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.742814 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.743029 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:30.743064 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:30.743092 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.743201 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.743220 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 15:27:30.746625 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.745849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36714 (local address 127.4.78.124:43347)
0924 15:27:30.745891 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.745893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.746082 (+ 189us) server_negotiation.cc:407] Connection header received
0924 15:27:30.746182 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.746183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.746195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.746225 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.746572 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36714: BlockingRecv error: recv got EOF from 127.0.0.1:36714 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:27:30.759671 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.756293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.4.78.125:42121)
0924 15:27:30.756332 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.756334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.756595 (+ 261us) server_negotiation.cc:407] Connection header received
0924 15:27:30.756695 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.756696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.756718 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.756749 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:30.756926 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.757642 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.758814 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.759100 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.759145 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.759421 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:30.759455 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:30.759484 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.759584 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.759602 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:27:30.763067 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.762256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36718 (local address 127.4.78.124:43347)
0924 15:27:30.762295 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.762297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.762475 (+ 178us) server_negotiation.cc:407] Connection header received
0924 15:27:30.762582 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.762583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.762596 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.762630 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.763003 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36718: BlockingRecv error: recv got EOF from 127.0.0.1:36718 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:27:30.783941 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.780516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56178 (local address 127.4.78.125:42121)
0924 15:27:30.780559 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.780562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.780877 (+ 315us) server_negotiation.cc:407] Connection header received
0924 15:27:30.780992 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.780993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.781004 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.781033 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:30.781245 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.782037 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.783161 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.783423 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.783466 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.783666 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:30.783699 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:30.783738 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.783843 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.783863 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:27:30.787513 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.786717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36734 (local address 127.4.78.124:43347)
0924 15:27:30.786761 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.786763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.786953 (+ 190us) server_negotiation.cc:407] Connection header received
0924 15:27:30.787051 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.787052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.787064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.787094 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.787458 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36734: BlockingRecv error: recv got EOF from 127.0.0.1:36734 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:27:30.824647 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.821094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.4.78.125:42121)
0924 15:27:30.821143 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.821146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.821371 (+ 225us) server_negotiation.cc:407] Connection header received
0924 15:27:30.821517 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.821518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.821529 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.821559 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:30.821798 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.822582 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.823806 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.824121 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.824198 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.824376 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:30.824410 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:30.824466 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.824541 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.824563 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 15:27:30.827370 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.826727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36742 (local address 127.4.78.124:43347)
0924 15:27:30.826768 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.826770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.826869 (+ 99us) server_negotiation.cc:407] Connection header received
0924 15:27:30.826933 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.826934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.826945 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.826996 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.827320 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36742: BlockingRecv error: recv got EOF from 127.0.0.1:36742 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:30.895479 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:30.892819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56188 (local address 127.4.78.125:42121)
0924 15:27:30.892888 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.892890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.893045 (+ 155us) server_negotiation.cc:407] Connection header received
0924 15:27:30.893150 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.893151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.893170 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.893296 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:30.893450 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.893967 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.894772 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:30.894967 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:30.895042 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:30.895204 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:30.895243 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:30.895303 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:30.895362 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 15:27:30.895383 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 15:27:30.897943 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:30.897297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36754 (local address 127.4.78.124:43347)
0924 15:27:30.897374 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:30.897376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:30.897419 (+ 43us) server_negotiation.cc:407] Connection header received
0924 15:27:30.897491 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:30.897492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:30.897503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:30.897559 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:30.897887 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36754: BlockingRecv error: recv got EOF from 127.0.0.1:36754 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:27:31.029285 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:31.027114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56204 (local address 127.4.78.125:42121)
0924 15:27:31.027184 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.027186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.027279 (+ 93us) server_negotiation.cc:407] Connection header received
0924 15:27:31.027335 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.027336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.027344 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.027410 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:31.027519 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.027995 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.028711 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.028883 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.028940 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:31.029072 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:31.029097 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:31.029134 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:31.029195 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 15:27:31.029224 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 15:27:31.031706 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:31.031097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36762 (local address 127.4.78.124:43347)
0924 15:27:31.031178 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.031180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.031190 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:31.031223 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.031224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.031234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.031291 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:31.031623 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36762: BlockingRecv error: recv got EOF from 127.0.0.1:36762 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 15:27:31.151268 27332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7357758fafb14783b9eff1dedbb0f4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7357758fafb14783b9eff1dedbb0f4ee" member_type: VOTER last_known_addr { host: "127.4.78.126" port: 44823 } } peers { permanent_uuid: "196add3fbb184da5842c43bb9480588d" member_type: VOTER last_known_addr { host: "127.4.78.125" port: 42121 } } peers { permanent_uuid: "095cc4c2e7ec485ea16a629aab2df407" member_type: VOTER last_known_addr { host: "127.4.78.124" port: 43347 } } }
I20260924 15:27:31.151424 27332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7357758fafb14783b9eff1dedbb0f4ee [sys.catalog]: This master's current role is: LEADER
I20260924 15:27:31.291249 27220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:31.289032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56214 (local address 127.4.78.125:42121)
0924 15:27:31.289099 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.289101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.289109 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:31.289137 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.289138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.289146 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.289221 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:31.289369 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.289857 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.290668 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.290852 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.290912 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:31.291039 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:31.291066 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:31.291104 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:31.291172 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:27:31.291188 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 15:27:31.293599 27295 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:27:31.293059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36770 (local address 127.4.78.124:43347)
0924 15:27:31.293125 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.293126 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.293134 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:27:31.293157 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.293157 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.293167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.293213 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:27:31.293557 (+ 344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36770: BlockingRecv error: recv got EOF from 127.0.0.1:36770 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:27:31.627223 27323 catalog_manager.cc:1259] Loaded cluster ID: f14d990529174ed7b5d0aaa346e5db9f
I20260924 15:27:31.627327 27323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407: loading cluster ID for follower catalog manager: success
I20260924 15:27:31.628134 27323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407: acquiring CA information for follower catalog manager: success
I20260924 15:27:31.628468 27323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 095cc4c2e7ec485ea16a629aab2df407: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:27:31.809504 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:31.807079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56226 (local address 127.4.78.125:42121)
0924 15:27:31.807240 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.807242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.807259 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:27:31.807342 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.807343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.807354 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.807414 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:31.807548 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.808050 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.808782 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.808975 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.809032 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:31.809196 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:31.809229 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:31.809282 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:31.809339 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 15:27:31.809366 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260924 15:27:31.814131 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:31.811451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.4.78.124:43347)
0924 15:27:31.811557 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.811559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.811569 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:27:31.811635 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.811635 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.811705 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.811782 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:31.811896 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.812547 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.813267 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.813482 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.813559 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:31.813718 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:31.813902 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:31.813999 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:31.814022 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 15:27:31.814043 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":52,"threads_started":1}
I20260924 15:27:31.818775 27484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:31.815749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.4.78.126:44823)
0924 15:27:31.816124 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.816127 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.816134 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:27:31.816178 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.816179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.816233 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.816331 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:31.816412 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.817034 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.817781 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.818134 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.818201 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:31.818350 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:31.818454 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:31.818511 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:31.818667 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 15:27:31.818691 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":51,"threads_started":1}
I20260924 15:27:31.818864 27473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:31.815737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.4.78.125:42121)
0924 15:27:31.815778 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.815780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.816008 (+ 228us) server_negotiation.cc:407] Connection header received
0924 15:27:31.816112 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.816113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.816122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.816203 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:31.816346 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.817093 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.818134 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.818408 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.818478 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:31.818597 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:31.818629 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:31.818680 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:31.818759 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 15:27:31.818781 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:27:31.819366 27479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:27:31.815737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.4.78.124:43347)
0924 15:27:31.815825 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 15:27:31.815827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:27:31.816005 (+ 178us) server_negotiation.cc:407] Connection header received
0924 15:27:31.816068 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:27:31.816069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:27:31.816080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:27:31.816127 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:27:31.816264 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.817058 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.818516 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:27:31.818798 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:27:31.818875 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:27:31.819127 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:27:31.819161 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:27:31.819205 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:27:31.819285 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:27:31.819307 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 15:27:31.820766 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 27073
I20260924 15:27:31.826588 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 27148
I20260924 15:27:31.831583 4409 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskwmKTEk/build/release/bin/kudu with pid 27223
2026-09-24T15:27:31Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2858 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5663 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (269478 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST